Test plan:
make test

also
 /home/pad/github/semgrep/semgrep-core/_build/default/bin/Main.exe -lang java -test_parse_lang /path/to/gradle/

to test the java parser on the open source gradle project gives
pretty good result, from 32 errors with just the Java pfff parser
to 11 errrors
Co-authored-by: Nathan Brahms <nathan@returntocorp.com>

Conflict appeared between commit e14ab83c, which rev'd the version of
mypy we were using on mainline code, and commit 2fb1d8e5, which added
a stripped-down mypy configuration for tests.

Here I rev the version of mypy used in tests.

However, doing this revealed some untyped declarations in
semgrep/evaluation.py, which are fixed here.

Flag will only affect logging/progress output to stderr

Closes https://github.com/returntocorp/semgrep/issues/1247

* Rename .bentoignore to .semgrepignore

See updated semgrep-action:
https://github.com/returntocorp/semgrep-action#ignoring-paths

* .github: Set deployment ID of semgrep job directly

This will get rid of the *** masking of every number 1 in github actions logs.

We currently do not include ocaml packaging/building tools in pypi sdist
release so users cannot actually install semgrep this way. Removing sdist
for now to avoid confusion

* pre-commit-config: Bump hook versions

* pre-commit-config: Use reorder-python-imports in py3.6+ mode

This way it'll update legacy imports automatically,
such as mock -> unittest.mock
and mypy_extensions.TypedDict -> typing.TypedDict.

Test plan:
make test
also semgrep-core -parse_ruby /path/to/rails gives the same
parsing stats (11 parsing errors), same with brew (9 parsing errors)
so the new code generator of martin does not introduce regressions for Ruby

Co-authored-by: Martin Jambon <Martin Jambon>

We want mypy to help us make sure we're calling methods correctly in
test code, to help us avoid unnecessary CI cycles. However, we don't
need to be strict with making sure we're defining types.

Here I add a second mypy config just for test code, with this less
restrictive configuration.
Co-authored-by: Drew Dennison <drew@returntocorp.com>

* Revise CONTRIBUTING.md and development.md

* Re-remove development.md

Test plan:
make test

test plan:
semgrep -l ruby now works

* Update dev docs, create semgrep-core/bin symlink

* Use latest ocaml-tree-sitter-lang

Co-authored-by: Martin Jambon <Martin Jambon>
Co-authored-by: Drew Dennison <drew@returntocorp.com>

* use consistant User Agent which includes the version

* * pfff: use latest, and CEs on AST generic changes

* Add token info from tree-sitter and add few test cases for semgrep/ruby

Test plan:
make test now should also run the tests under semgrep-core/tests/ruby/

* Also get deep metavars for IdSpecial (namely, ==)

Tested with make test

* Added note to install pkg-config in development instructions (#1202)

* added a line to install pkg-config due to opam install --deps-only -y failure

* two small updates to the Run instructions

* Fix #1191, #438, use '/src' as Docker code volume (#1192)

* Support Assign

Options: in addition to exact match, dots, and metavar replacement, we also recursively generalize the right hand side

Tested with _build/default/bin/Main.exe -synthesize_patterns 8:3-8:56 tests/SYNTHESIZING/set_cookie.py

However, doesn't currently have a working test case

* pfff: bring in python2 parse fixes (#1205)

* pfff: bring in python2 parse fixes

- Parse python files with trailing whitespace
- Parse python2 tuple as parameter in function/lambda definition
- Parse python3.8 positional only parameters (PEP 570)
- Parse python2 implicit array in comprehension

* Revert pfff to c554bc0cb3adb67556bf8bf1eb19e60a033ec9d1

e419abdf7aad79f0b04f5d5ff349081fb1cab7fa had equivalence regressions
that make semgrep-core fail tests

* Release 0.14.0b1

* Release 0.14.0

* Added test for new equal feature

Currently this test does not pass. See behavior with "make test". Note that the error message is somewhat misleading, since pattern_to_string is used to output the code it should match. In reality, the code it should match is as specified by range and file. In this case, it is

 resp = r.set_cookie("sessionid", resp, "RANDOM-UUID")

(though of course import resolution must be taken into account)

* Handle file-not-found errors more gracefully (#1207)

* Handle file-not-found errors more gracefully

Re-factored target manager init method to use attr.s for clarity and passed output_handler to target_manager.

* Fix test failures from reordered arguments to TargetManager

* update CHANGELOG

* docs: Fix pre-commit instructions (#1220)

* WIP: Ruby tree-sitter integration (#1127)

* Require a more recent version of dune, used to compile C and C++ files
from tree-sitter.

* Add submodule ocaml-tree-sitter-lang

* Use latest Ruby boilerplate

* Use latest ocaml-tree-sitter-lang

* Bento-ignore new submodule ocaml-tree-sitter-lang

* Add trailing slash to bento-ignored directory

* Add setup for ocaml-tree-sitter

* Use latest ocaml-tree-sitter

* Fixes for ocaml-tree-sitter setup

* Add missing step to workflow

* Connecting to add_lang branch in pfff adding Ruby and Typescript in Lang.ml

Test plan:
make
make test

* * semgrep-core/bin/Main.ml: new -parse_ruby action testing tree-sitter parsers

* * semgrep-core/bin/Main.ml: run parser in its own process, to avoid
segfaults

test plan:
Main.exe -parse_ruby ~/work/lang-ruby
which is a big corpus of ruby projects (rails, gitlab) does not
segfault anymore and return pretty good parsing stats
(97% on rails). Far better than the Ruby parser in pfff which
can deal with only 15% of modern rails code.

* Factorize code in Parse_code.ml

This will help the gradual transition to tree-sitter parsers.
We can now in Parse_code.ml call not only pfff parser but also tree-sitter
parsers

Test plan:
make test

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: Skeleton code

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: just copy boilerplate

Now needs to fill it ...

Test plan:
pad@yrax:~/semgrep$ yy -dump_ast tests/ruby/foo.rb
+ /home/pad/github/semgrep/semgrep-core/_build/default/bin/Main.exe -dump_ast tests/ruby/foo.rb
(((Stmts_rep1_choice_stmt_term_opt_stmt
   (((Stmt_term
      ((Stmt_exp
        (Exp_cmd_call
         (Cmd_call_choice_var_cmd_arg_list
          ((Var (Id puts))
           (Cmd_arg_list_arg_rep_COMMA_arg
            ((Arg_arg
              (Arg_prim
               (Prim_str ("\"" (((Str_content "Hello World"))) "\""))))
             ()))))))
       (Term_line_brk ""))))
    ())))
 ())
Fatal error: exception (Failure "not implemented")
Raised at file "stdlib.ml", line 29, characters 22-33
Called from file "parsing/Parse_ruby_tree_sitter.ml", line 1142, characters 20-30
Called from file "parsing/Parse_ruby_tree_sitter.ml", line 216, characters 23-35
Called from file "list.ml", line 92, characters 20-23
Called from file "parsing/Parse_ruby_tree_sitter.ml", line 213, characters 8-258
Called from file "parsing/Parse_ruby_tree_sitter.ml", line 1660, characters 16-28
Called from file "parsing/Parse_code.ml", line 29, characters 16-49
Called from file "parsing/Parse_code.ml", line 38, characters 12-42
Called from file "bin/Main.ml", line 773, characters 12-48
Called from file "Common.ml", line 1294, characters 12-16
Re-raised at file "Common.ml", line 1292, characters 8-302
Called from file "Common.ml", line 1262, characters 45-49
Called from file "Common.ml", line 82, characters 14-18
Re-raised at file "Common.ml", line 87, characters 10-11
Called from file "Common.ml", line 1259, characters 6-10
Called from file "bin/Main.ml", line 1134, characters 2-55

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: handling basic
puts "Hello World"

* * semgrep-core/tests/ruby/foo.rb: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * pfff: using latest from add_lang with ast_ruby modifications

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress before I stop
for today

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: misc

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * pfff: use latest add_lang

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: no more Todo! v0!

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: use latest pfff

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs new ast_ruby.ml

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: use Char|Complex|Rational

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: use new variable type

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: indent

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs using latest ast_ruby.ml

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs using latest, DotAccess

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* Update ocaml-tree-sitter

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs recscue_clause, and no
more 'fk'!

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs Block and bracket

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs option2 removal

* * pfff: use latest!

* Fix incorrect build commands for ocaml-tree-sitter's runtime.

* Fix dependency setup

* Use latest ocaml-tree-sitter

* Run new configure step in ocaml-tree-sitter install

* Use alpine instead of debian+musl

* Update ocaml-tree-sitter

* Show commit info

* Fix non-exhaustive pattern match

* Update submodule pfff

* Switch to alpine for all ocaml builds in github actions.

* Try to set up python for alpine

* Force static linking on alpine

* Try using python 3.8

* Split benchmark.yml into two alpine/ocaml and debian/python

* Fix config

* Add missing copy to export semgrep-core

* Factor out the semgrep-core build

* Add optional 'git clone' for convenience

* Update pfff

* Update pfff

* Split python-dependent tests from semgrep-core build

* Restore the bizarre 'opam env --root /home/opam/.opam --set-root'

* Update ocaml-tree-sitter

* Make ubuntu release reuse the semgrep-core binary.

* Add missing checkout step

* Fix this and that

* * pfff: use latest add_lang, merged

* use latest pfff add_lang branch and CEs

* Fix dockerfile

* Add a root makefile that supports 'make clean'

* Add cleanup step before starting the build.

* * pfff: use latest

* * semgrep-core/bin/Main.ml: Fix possible regression introducing when merging
develop in tree-sitter.

Probably during some past merge, we used Parse_generic.parse_with_lang
which does not resolve name instead of the correct (but easy
to mispell) Parse_code.parse_with_lang.
This meant many tests were failing because we were not resolving names
anymore.

Test plan:
make test in semgrep now work

* * semgrep-core/tests/python/misc_faketok3.py: fix regression

* * Dockerfile: shutup the warning about sudo

There is probably a better way to fix this ... but right now
I really want to merge the tree-sitter branch in develop so
this will have to wait.
@brendon or @matt for help fixing that.
Co-authored-by: pad <yoann.padioleau@gmail.com>
Co-authored-by: Martin Jambon <Martin Jambon>

* output: Fix SARIF format (#1219)

Changes were tested by running the new SARIF snapshot file through the
validator at https://sarifweb.azurewebsites.net/Validation

Closes https://github.com/returntocorp/semgrep/issues/1209



* Use latest version of different submodules (#1222)

Test plan:
make; make test

* Also get deep metavars for IdSpecial (namely, ==)

Tested with make test

* Support Assign

Options: in addition to exact match, dots, and metavar replacement, we also recursively generalize the right hand side

Tested with _build/default/bin/Main.exe -synthesize_patterns 8:3-8:56 tests/SYNTHESIZING/set_cookie.py

However, doesn't currently have a working test case

* Added test for new equal feature

Currently this test does not pass. See behavior with "make test". Note that the error message is somewhat misleading, since pattern_to_string is used to output the code it should match. In reality, the code it should match is as specified by range and file. In this case, it is

 resp = r.set_cookie("sessionid", resp, "RANDOM-UUID")

(though of course import resolution must be taken into account)

* Commented out the failing test

Added print statement to show the AST of the patterns.
Co-authored-by: Clara McCreery <clara@returntocorp.com>
Co-authored-by: mschwager <mschwager@users.noreply.github.com>
Co-authored-by: Brendon Go <brendon.go@gmail.com>
Co-authored-by: Clara McCreery <chmccreery@gmail.com>
Co-authored-by: Bence Nagy <bence@underyx.me>
Co-authored-by: Martin Jambon <martin@mjambon.com>
Co-authored-by: pad <yoann.padioleau@gmail.com>
Co-authored-by: Yoann Padioleau <pad@returntocorp.com>

* Populate the root makefile with useful targets, most notably 'make rebuild'.

* Fix the python build step

Co-authored-by: Martin Jambon <Martin Jambon>

Test plan:
make; make test

Changes were tested by running the new SARIF snapshot file through the
validator at https://sarifweb.azurewebsites.net/Validation

Closes https://github.com/returntocorp/semgrep/issues/1209

* Require a more recent version of dune, used to compile C and C++ files
from tree-sitter.

* Add submodule ocaml-tree-sitter-lang

* Use latest Ruby boilerplate

* Use latest ocaml-tree-sitter-lang

* Bento-ignore new submodule ocaml-tree-sitter-lang

* Add trailing slash to bento-ignored directory

* Add setup for ocaml-tree-sitter

* Use latest ocaml-tree-sitter

* Fixes for ocaml-tree-sitter setup

* Add missing step to workflow

* Connecting to add_lang branch in pfff adding Ruby and Typescript in Lang.ml

Test plan:
make
make test

* * semgrep-core/bin/Main.ml: new -parse_ruby action testing tree-sitter parsers

* * semgrep-core/bin/Main.ml: run parser in its own process, to avoid
segfaults

test plan:
Main.exe -parse_ruby ~/work/lang-ruby
which is a big corpus of ruby projects (rails, gitlab) does not
segfault anymore and return pretty good parsing stats
(97% on rails). Far better than the Ruby parser in pfff which
can deal with only 15% of modern rails code.

* Factorize code in Parse_code.ml

This will help the gradual transition to tree-sitter parsers.
We can now in Parse_code.ml call not only pfff parser but also tree-sitter
parsers

Test plan:
make test

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: Skeleton code

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: just copy boilerplate

Now needs to fill it ...

Test plan:
pad@yrax:~/semgrep$ yy -dump_ast tests/ruby/foo.rb
+ /home/pad/github/semgrep/semgrep-core/_build/default/bin/Main.exe -dump_ast tests/ruby/foo.rb
(((Stmts_rep1_choice_stmt_term_opt_stmt
   (((Stmt_term
      ((Stmt_exp
        (Exp_cmd_call
         (Cmd_call_choice_var_cmd_arg_list
          ((Var (Id puts))
           (Cmd_arg_list_arg_rep_COMMA_arg
            ((Arg_arg
              (Arg_prim
               (Prim_str ("\"" (((Str_content "Hello World"))) "\""))))
             ()))))))
       (Term_line_brk ""))))
    ())))
 ())
Fatal error: exception (Failure "not implemented")
Raised at file "stdlib.ml", line 29, characters 22-33
Called from file "parsing/Parse_ruby_tree_sitter.ml", line 1142, characters 20-30
Called from file "parsing/Parse_ruby_tree_sitter.ml", line 216, characters 23-35
Called from file "list.ml", line 92, characters 20-23
Called from file "parsing/Parse_ruby_tree_sitter.ml", line 213, characters 8-258
Called from file "parsing/Parse_ruby_tree_sitter.ml", line 1660, characters 16-28
Called from file "parsing/Parse_code.ml", line 29, characters 16-49
Called from file "parsing/Parse_code.ml", line 38, characters 12-42
Called from file "bin/Main.ml", line 773, characters 12-48
Called from file "Common.ml", line 1294, characters 12-16
Re-raised at file "Common.ml", line 1292, characters 8-302
Called from file "Common.ml", line 1262, characters 45-49
Called from file "Common.ml", line 82, characters 14-18
Re-raised at file "Common.ml", line 87, characters 10-11
Called from file "Common.ml", line 1259, characters 6-10
Called from file "bin/Main.ml", line 1134, characters 2-55

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: handling basic
puts "Hello World"

* * semgrep-core/tests/ruby/foo.rb: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * pfff: using latest from add_lang with ast_ruby modifications

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress before I stop
for today

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: misc

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * pfff: use latest add_lang

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: progress

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: no more Todo! v0!

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: use latest pfff

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs new ast_ruby.ml

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: use Char|Complex|Rational

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: use new variable type

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: more

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: indent

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs using latest ast_ruby.ml

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs using latest, DotAccess

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* Update ocaml-tree-sitter

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs recscue_clause, and no
more 'fk'!

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs Block and bracket

* * semgrep-core/parsing/Parse_ruby_tree_sitter.ml: CEs option2 removal

* * pfff: use latest!

* Fix incorrect build commands for ocaml-tree-sitter's runtime.

* Fix dependency setup

* Use latest ocaml-tree-sitter

* Run new configure step in ocaml-tree-sitter install

* Use alpine instead of debian+musl

* Update ocaml-tree-sitter

* Show commit info

* Fix non-exhaustive pattern match

* Update submodule pfff

* Switch to alpine for all ocaml builds in github actions.

* Try to set up python for alpine

* Force static linking on alpine

* Try using python 3.8

* Split benchmark.yml into two alpine/ocaml and debian/python

* Fix config

* Add missing copy to export semgrep-core

* Factor out the semgrep-core build

* Add optional 'git clone' for convenience

* Update pfff

* Update pfff

* Split python-dependent tests from semgrep-core build

* Restore the bizarre 'opam env --root /home/opam/.opam --set-root'

* Update ocaml-tree-sitter

* Make ubuntu release reuse the semgrep-core binary.

* Add missing checkout step

* Fix this and that

* * pfff: use latest add_lang, merged

* use latest pfff add_lang branch and CEs

* Fix dockerfile

* Add a root makefile that supports 'make clean'

* Add cleanup step before starting the build.

* * pfff: use latest

* * semgrep-core/bin/Main.ml: Fix possible regression introducing when merging
develop in tree-sitter.

Probably during some past merge, we used Parse_generic.parse_with_lang
which does not resolve name instead of the correct (but easy
to mispell) Parse_code.parse_with_lang.
This meant many tests were failing because we were not resolving names
anymore.

Test plan:
make test in semgrep now work

* * semgrep-core/tests/python/misc_faketok3.py: fix regression

* * Dockerfile: shutup the warning about sudo

There is probably a better way to fix this ... but right now
I really want to merge the tree-sitter branch in develop so
this will have to wait.
@brendon or @matt for help fixing that.
Co-authored-by: pad <yoann.padioleau@gmail.com>
Co-authored-by: Martin Jambon <Martin Jambon>

* Handle file-not-found errors more gracefully

Re-factored target manager init method to use attr.s for clarity and passed output_handler to target_manager.

* Fix test failures from reordered arguments to TargetManager

* update CHANGELOG

Release 0.14.0