- Make 'make bump' run with GNU sed.
- Remove the patching of documentation file that moved to separate repo
  https://github.com/returntocorp/semgrep-docs/blob/main/docs/integrations.md.

Co-authored-by: Martin Jambon <Martin Jambon>

* Add ability to define unit tests that check the matched contents rather
than just the number of matches.

* Fix incorrect matching of trailing dots. They now cover up to 10 lines
even if the end of the document isn't reached.

Co-authored-by: Martin Jambon <Martin Jambon>

* Improve heuristic for skipping binary files.

* Add tests for gibberish file detection.

Co-authored-by: Martin Jambon <Martin Jambon>

* Add '--force' option to spacegrep so as to not skip gibberish files.

* Update bug report URL.

Co-authored-by: Martin Jambon <Martin Jambon>

* Allow spacecat to read from files or directories instead of stding.

* Fix off-by-one error in line count, leading to one-line files being
treated as gibberish (non-text) and ignored. A warning is now printed on
stderr if a file is ignored.

* Print one blank line between matches, not systematically after a match.

Co-authored-by: Martin Jambon <Martin Jambon>

* Remove semgrep-core/finding/

The file filtering is now handled in the Python wrapper so
we don't need this code anymore. Morever the code uses
the Dune.glob library which conflicts with the LSP library, so
time to remove it anyway.

test plan:
make

* Uncomment the commented LSP code now that LSP is in OPAM

test plan:
make
make test

* Use latest pfff fixing OCaml parse error regarding GADTs

Test plan:
make check now works

* Upgrade to a more recent base image in Dockerfile.

Co-authored-by: Martin Jambon <Martin Jambon>

* Avoid creating empty blocks when parsing matching braces.
Add failing test "indented ellipsis".

* Fix matching bug. An indented block containing just an ellipsis can now
match an empty sequence rather than requiring to match a block.

* Fix spacegrep matching bug regarding ellipses: the element matched right
after an ellipsis was unchecked for its location, allowing it to be
further than 10 lines down. This commit also fixes the behavior of
multiple dots e.g. '... ...' used to skip more than 10 lines.

Co-authored-by: Martin Jambon <Martin Jambon>

* Avoid creating empty blocks when parsing matching braces.
Add failing test "indented ellipsis".

* Fix matching bug. An indented block containing just an ellipsis can now
match an empty sequence rather than requiring to match a block.

Co-authored-by: Martin Jambon <Martin Jambon>

* updating perf baseline to 0.29.0

* switching back to 5% slower as compared to 10%
Co-authored-by: Colleen Dai <chai@Colleens-MBP.attlocal.net>

This pull request deletes `docs/` and directs readers to https://semgrep.dev/docs. It also shortens the README in favor of documentation on the external site, reducing duplicated materials and out of date information.

test plan:
make test

A recurring problem with identifiers vs expressions.
Fixes https://github.com/returntocorp/semgrep/issues/1861

test plan:
test file included

We want assignments to match over field definitions in classes
which use the '=' syntax.
We do not want them to match field definitions in objects
which use the ':' syntax.

Fixes https://github.com/returntocorp/semgrep/issues/1846

test plan:
test file included
make test

Fixes https://github.com/returntocorp/semgrep/issues/1902

test plan:
test files included

Release 0.29.0

Co-authored-by: Bence Nagy <bence@returntocorp.com>

Co-authored-by: Bence Nagy <bence@returntocorp.com>

* Cleanup AST Generic

test plan:
make
make test

* * semgrep-core/pfff: use latest

* Support dynamic class names in generic AST, for Ruby

This is the first step for https://github.com/returntocorp/semgrep/issues/1892

test plan:
make test

* use latest pfff

* Test cases for metavariable matching complex class names

Fixes https://github.com/returntocorp/semgrep/issues/1892

test plan:
test file included

spacegrep. These examples are used to populate the cheatsheet on the
semgrep.dev website. The 'GENERIC' folder was renamed to 'POLYGLOT'
to avoid confusion and problems on case-insensitive file systems.

Co-authored-by: Martin Jambon <Martin Jambon>

* Upgrade javascript and typescript grammar, both from tree-sitter.

* Adapt to new javascript and typescript grammars (WIP)

* Adapt to new javascript and typescript grammars (part 2/2).

Co-authored-by: Martin Jambon <Martin Jambon>

- Allow user to specify registry path without having to add semgrep.dev url
  i.e.: instead of `--config https://semgrep.dev/p/r2c-ci` users can use `--config p/r2c-ci`
- Allow user to specify snippet id withouth having to add semgrep.dev url
  i.e.: instead of `--config https://semgrep.dev/s/username:snippetname`
  users can use `--config username:snippetname`

* Make README language support consistent with https://dashboard.semgrep.dev/languages

Co-authored-by: Luke O'Malley <luke@returntocorp.com>

* Error out in --test if ruleid or ok not in reported IDs

* Check for results without accompanying tests too

Also modified autofix tests to actually verify inline autofix

Fixes https://github.com/returntocorp/semgrep/issues/1841

test plan:
test file included
make
make test
also tried in semgrep and against semgrep-rules and no regressions

Fixes
https://github.com/returntocorp/semgrep/issues/1877

test plan:
test file included

Nobody needs the internal semgrep libraries, so switch
from public_name to name in the dune files, which reduce the
size of make install

test plan:
make
make test

* First step in leveraging tree-sitter error recovery

This is just the first step to help
https://github.com/returntocorp/semgrep/issues/1879

test plan:
make
make test

* Step 2: Move code out of Parse_tree_sitters_helpers in Parse_code

This move the code to the right place where we can leverage the
returned list of errors.
This also helps https://github.com/returntocorp/semgrep/issues/1879

test plan:
make
make test

* Step 3: handle Partial result in Parse_code.ml

test plan:
$ /home/pad/semgrep/_build/default/cli/Main.exe -lang js -e foo tests/OTHER/skipping_errors/test.js
[0.086  Info       Main.Parse_code      ] trying to parse with TreeSitter parser tests/OTHER/skipping_errors/test.js
[0.086  Info       Main.Parse_code      ] partial error (Parse_info.Parsing_error(_)) with TreeSitter parser
[0.087  Info       Main.Parse_code      ] trying to parse with Pfff parser tests/OTHER/skipping_errors/test.js
[0.087  Info       Main.Parse_code      ] running the pfff JS parser with 5s timeout
[0.087  Debug      Main.Parse_code      ] exn (Parse_info.Parsing_error(_)) with Pfff parser
[0.087  Debug      Main.Parse_code      ] exn again ((Failure "no parser found for tests/OTHER/skipping_errors/test.js")) but return original exn (Parse_info.Parsing_error(_))
[0.087  Debug      Main.Parse_code      ] exn again (Parse_info.Parsing_error(_)) but return Partial
tests/OTHER/skipping_errors/test.js:2
     foo();

but we should also return errors (final step in next PR)

* Final step: propagate recoved parsing errors to the top

Test plan:
See how below we now both print matched code and warn about parsing errors

$ /home/pad/semgrep/_build/default/cli/Main.exe -lang js -e foo tests/OTHER/skipping_errors/test.js
[0.109  Info       Main.Parse_code      ] trying to parse with TreeSitter parser tests/OTHER/skipping_errors/test.js
[0.110  Info       Main.Parse_code      ] partial error (Parse_info.Parsing_error(_)) with TreeSitter parser
[0.110  Info       Main.Parse_code      ] trying to parse with Pfff parser tests/OTHER/skipping_errors/test.js
[0.110  Info       Main.Parse_code      ] running the pfff JS parser with 5s timeout
[0.110  Debug      Main.Parse_code      ] exn (Parse_info.Parsing_error(_)) with Pfff parser
[0.110  Debug      Main.Parse_code      ] exn again ((Failure "no parser found for tests/OTHER/skipping_errors/test.js")) but return original exn (Parse_info.Parsing_error(_))
[0.110  Debug      Main.Parse_code      ] exn again (Parse_info.Parsing_error(_)) but return Partial
WARNING: fail to fully parse tests/OTHER/skipping_errors/test.js
tests/OTHER/skipping_errors/test.js:2
     foo();

$  semgrep -l js -e foo test.js --verbose
semgrep warn: parse error
  --> test.js:6
6 |     return 1 %%% 2;
  |               ^^
= help: If the code appears to be valid, this may be a semgrep bug.
Could not parse test.js as js

test.js
2:    foo();
ran 1 rules on 1 files: 1 findings
1 files could not be analyzed; run with --verbose for details or run with --strict to exit non-zero if any file cannot be analyzed