- 17 Aug, 2021 4 commits
-
-
brendon authored
-
Brendon Go authored
* Handle empty pattern error * fixup! Handle empty pattern error
-
Iago Abal authored
And added an optimized `Pattern_match.uniq` function to replace `Common.uniq_by`. Note that Semgrep returns the same matches as before, but sometimes in a different order. This issue was reported by a customer weeks ago but we could not figure it out without an example. I just happened to hit the same bug when benchmarking some taint rules. test plan: % make test % semgrep-core -timeout 5 -j 1 -profile -lang js \ -config semgrep-rules/typescript/react/security/audit/react-props-injection.yaml \ semgrep/parsing-stats/lang/javascript/tmp/mui-org-material-ui/test/bundling/fixtures/next-webpack5 #^ Try it several times, it no longer triggers a fatal error Rule react-props-injection contains the following pattern: import $PROPS from "..."; ... This pattern leads to more than 90k matches in lang/javascript/tmp/mui-org-material-ui/test/bundling/fixtures/next-webpack5/pages/next-webpack.fixture.js which causes Common.uniq_by to "explode". If we are unlucky and the analysis timesout while we are testing the equality of two ASTs, then `AST_utils.busy_with_equal` is left in the wrong state and all the subsequent equality tests will lead to a fatal error.
-
Yoann Padioleau authored
test plan: make test
-
- 14 Aug, 2021 1 commit
-
-
Yoann Padioleau authored
test plan: make test
-
- 13 Aug, 2021 2 commits
-
-
Yoann Padioleau authored
* [C++] more Parse_cpp_tree_sitter.ml todos This will help PA-14 test plan: make test * progress * fix CI
-
mschwager authored
Fixes #3680.
-
- 12 Aug, 2021 1 commit
-
-
mschwager authored
Fixes #3705.
-
- 11 Aug, 2021 4 commits
-
-
David Frankel authored
* Combine else-if tokens * Add type param support * [Hack] Add basic testing * Finish alpha features * Add more tests (still need to update semgrep-hack ref) * Improve dots_params * [Hack] Extend metavariables * Update semgrep-hack version * Add hacklang to semgrep targets * Update error messages
-
kingbbello authored
* adding more options to semgrep sarif formatter * add changelog * handle empty tag casse
-
Iago Abal authored
* Optimize pattern $X Fixes: bfc4f3da ("Revert `pattern: $X` optimization (#3478)") Fixes: 32e88975 ("Separate pattern: $X in anded patterns to a special field (#3435)") test plan: % make test # tests included % semgrep-core -lang py -config \ ~/semgrep/semgrep-core/tests/OTHER/rules/pattern-x-1.yaml \ bench/django/input/django #^ now takes ~6 seconds (i.e., almost 10x faster) * engine: Clean up Specialize_formula test plan: make test * Emma's comments
-
Martin Jambon authored
* Ensure that paths aren't directories Workaround for issue with git submodules and git ls-files, see #3660 for details. * Add submodule fix to changelog * Run pre-commit hooks * Ensure we don't end up with folders in a list of targets * Update CHANGELOG.md
-
- 10 Aug, 2021 4 commits
-
-
raghavjain3 authored
-
Yoann Padioleau authored
-
Yoann Padioleau authored
progress test plan: make test
-
Emma Jin authored
-
- 09 Aug, 2021 7 commits
-
-
Brian Kroth authored
* fixups and tests for running benchmarks in docker * need to pass --quiet to avoid logging info from failing to json parse * Update perf/run-benchmarks
-
Iago Abal authored
- Add backtrace to fatal errors to help diagnosing bug reports (Pfff). - Report rule evaluation errors instead of "silently" logging them. Helps #3547 test plan: % cat fatal.py def foo(): return 1 } % semgrep -l py -e 'x' fatal.py running 1 rules... semgrep-core reported a fatal error: ----- Fatal Error: (Failure "Lexer_python.top_mode: empty stack") Raised at Parse_target.run in file "src/parsing/Parse_target.ml", line 176, characters 17-26 ... ----- Please file a bug report at https://github.com/returntocorp/semgrep/issues/new/choose ... % cat warn.yaml rules: - id: test languages: [python] patterns: - pattern: $X - metavariable-pattern: metavariable: $Y pattern: x message: Test severity: ERROR % cat warn.py x % semgrep -c warn.yaml -v warn.py ... semgrep-core reported a matching error --> matching internal error: rule test: metavariable-pattern failed because $Y it not in scope, please check your rule ...
-
Yoann Padioleau authored
-
Yoann Padioleau authored
This will help https://github.com/returntocorp/semgrep/issues/3576 test plan: make test
-
Yoann Padioleau authored
-
Yoann Padioleau authored
This closes https://github.com/returntocorp/semgrep/issues/3669 This is a bit ugly. The right fix would be to extend the AST_generic.expr record to store the leftmost and rightmost token in the expr, so we would not need to abuse tok in the expr_kind itself. test plan: test file included
-
Isaac Evans authored
-
- 08 Aug, 2021 1 commit
-
-
Yoann Padioleau authored
* [OCaml] support open XXx entity aliasing by using LSP test plan: ``` yy -no_bloom_filter -lsp -lang ocaml -e 'AST_generic.fake_bracket $X' tests/test_lsp.ml -log_config_file /tmp/xxx + /home/pad/yy/_build/default/src/cli/Main.exe -no_bloom_filter -lsp -lang ocaml -e 'AST_generic.fake_bracket $X' tests/test_lsp.ml -log_config_file /tmp/xxx START tests/test_lsp.ml:17 AST_generic.fake_bracket [] |> ignore; tests/test_lsp.ml:18 G.fake_bracket [] |> ignore; "AST_generic" tests/test_lsp.ml:19 fake_bracket [] |> ignore; yy -no_bloom_filter -lsp -lang ocaml -e 'AST_generic.Call ($X, $Y)' tests/test_lsp.ml -log_config_file /tmp/xxx + /home/pad/yy/_build/default/src/cli/Main.exe -no_bloom_filter -lsp -lang ocaml -e 'AST_generic.Call ($X, $Y)' tests/test_lsp.ml -log_config_file /tmp/xxx START tests/test_lsp.ml:6 let res0 = AST_generic.Call (Int (None, fake ""), fb []) in tests/test_lsp.ml:6 let res0 = AST_generic.Call (Int (None, fake ""), fb []) in tests/test_lsp.ml:7 let res1 = G.Call (Int (None, fake ""), fb []) in tests/test_lsp.ml:7 let res1 = G.Call (Int (None, fake ""), fb []) in "AST_generic" tests/test_lsp.ml:8 let res2 = Call (Int (None, fake ""), fb []) in "AST_generic" tests/test_lsp.ml:8 let res2 = Call (Int (None, fake ""), fb []) in yy -no_bloom_filter -lsp -lang ocaml -e '| AST_generic.Call ($X, $Y)' tests/test_lsp.ml -log_config_file /tmp/xxx + /home/pad/yy/_build/default/src/cli/Main.exe -no_bloom_filter -lsp -lang ocaml -e '| AST_generic.Call ($X, $Y)' tests/test_lsp.ml -log_config_file /tmp/xxx START tests/test_lsp.ml:10 | AST_generic.Call (x, (_, [], _)) -> 1 tests/test_lsp.ml:11 | G.Call (x, (_, [_], _)) -> 1 "AST_generic" tests/test_lsp.ml:12 | Call (x, y) -> 1 ``` * misc
-
- 06 Aug, 2021 5 commits
-
-
Yoann Padioleau authored
* Refactor m_name, factorize more code test plan: make test * more * more * factorize m_type naming part part in m_name * factorize m_attr
-
Yoann Padioleau authored
We should be able to factorize more code and move more aliasing logic in m_name, for TyN and NamedAttr too (and maybe m_expr). test plan: test file included
-
Iago Abal authored
test plan: 1. make test # tests included 2. semgrep -l py -e '42' any_file.py
-
Yoann Padioleau authored
This also factorize a bit name resolution for all 'name' types. test plan: ``` pad@yrax yy (naming_ctor)]$ yy -lang ocaml -dump_named_ast tests/ocaml/aliasing_qualified_contructor.ml + /home/pad/yy/_build/default/src/cli/Main.exe -lang ocaml -dump_named_ast tests/ocaml/aliasing_qualified_contructor.ml [0.038 Info Main.Dune__exe__Main ] loaded log_config.json [0.038 Info Main.Dune__exe__Main ] Executed as: /home/pad/yy/_build/default/src/cli/Main.exe -lang ocaml -dump_named_ast tests/ocaml/aliasing_qualified_contructor.ml [0.038 Info Main.Dune__exe__Main ] Version: semgrep-core version: v0.61.0-9-g14616953-dirty, pfff: 0.42 [0.038 Info Main.Parse_target ] trying to parse with Pfff parser tests/ocaml/aliasing_qualified_contructor.ml [0.038 Info Main.Parse_target ] Parse_target.parse_and_resolve_name_use_pfff_or_treesitter done Pr( [DefStmt( ({ name=EN( Id(("G", ()), { id_resolved=Ref(Some((ImportedModule( DottedName([("AST_generic", ())])), 1))); id_type=Ref(None); id_constness=Ref(None); })); attrs=[]; tparams=[]; }, ModuleDef({mbody=ModuleAlias([("AST_generic", ())]); }))); DefStmt( ({ name=EN( Id(("foo", ()), {id_resolved=Ref(None); id_type=Ref(None); id_constness=Ref(None); })); attrs=[]; tparams=[]; }, FuncDef( {fkind=(Function, ()); fparams=[ParamPattern(PatLiteral(Unit(())))]; frettype=None; fbody=OtherStmt(OS_ExprStmt2, [E( Constructor( IdQualified( (("Call", ()), {name_qualifier=Some(QDots([("G", ())])); name_typeargs=None; }), { id_resolved=Ref(Some((ImportedEntity( [("AST_generic", ()); ("Call", ())]), 0))); id_type=Ref(None); id_constness=Ref(None); }), [L(Int((Some(1), ()))); L(Int((Some(2), ())))]))]); })))]) ```
-
Martin Jambon authored
* Open files in binary mode so as to bypass CRLF translation on Windows. * Update pfff * Update pfff * Update pfff
-
- 05 Aug, 2021 6 commits
-
-
Iago Abal authored
The GC alarm doesn't trigger so reliably it seems, the test was failing on my laptop.
-
Brendon Go authored
-
Brendon Go authored
-
Yoann Padioleau authored
This will allow later to perform aliasing also on constructor calls, e.g. finding 'G.Call (foo, bar)' when looking for AST_generic.Call($X, $Y) This PR just do the Constructor refactoring. test plan: make
-
Yoann Padioleau authored
It is quite common in OCaml (at least in my OCaml codebases), to use module aliasing such as 'module G = AST_generic'. We want that a pattern like 'AST_generic.foo' also match code where the AST_generic module has been aliased, just like we do in Python or Go where we handle import aliasing. test plan: test file included
-
Yoann Padioleau authored
The main change is that we just skip big files. They cause too much problems and anyway those files are not real, they are autogenerated, or test files for benchmarks, or just huge data converted as code (e.g., unicode tables). Do we do similar filtering in semgrep itself? I remember martin at some point was looking at the size of files in spacegrep. test plan: make test ./run-lang c
-
- 04 Aug, 2021 4 commits
-
-
Brendon Go authored
Release 0.61.0
-
mschwager authored
-
Matt Schwager authored
-
Yoann Padioleau authored
* Many fixes to improve the parsing_stat CI job The Lua parsing stat job was failing because it was using a non-github repo which triggered an RPC error at clone time. The Kotlin parsing stat job was failing because it was using CST.dump_tree which output things on stdout, which then messes up the json. The C parsing stat job was failing because of some out of memory fatal errors on huge files. I tried to use Memory_limit but ran in many issues. It's hard to intercept those Out_of_memory reliably. test plan: ./run-lang --upload lua ./run-lang --upload kotlin ./run-lang --upload c * iago's comment
-
- 03 Aug, 2021 1 commit
-
-
Emma Jin authored
* Remove location information in cli errors with temp files Since the temp file error message is not helpful, remove it. Test plan: Modify `metavar_pattern_lang.yaml` in tests/OTHER/rules to have the pattern `bad eval_C("$CODE")`. Then run ``` (semgrep) ➜ rules git:(develop) ✗ semgrep --config metavar_pattern_lang.yaml metavar_pattern_lang.py running 1 rules... semgrep error: invalid pattern Pattern `bad eval_C("$CODE")` could not be parsed as a Python semgrep pattern ``` * Updated snapshots * Changelog
-