DryRun Security vs. Snyk, CodeQL, SonarQube, and Semgrep – Python (Django) Security Analysis Showdown

Context is King in Python Django Application Security

In this installment of our head-to-head series, we take on a Python Django application—the Vulnerable Task Manager (VTM)—to see how DryRun Security stacks up against traditional SAST tools: Snyk Code, GitHub CodeQL, SonarQube, and Semgrep. This is the third in a series of head-to-head tests. Previously we tested vulnerable code in RailsGoat and C# against the same tools.

For this blog post, the test code was introduced in PR #5 of our public DryRunSecuritySandbox repo, so you can review everything firsthand. This pull request deliberately injects several security bugs into the Django app, simulating a mix of obvious and nuanced vulnerabilities.

The results were eye-opening: DryRun Security contextual security analysis caught every vulnerability, including tricky authorization logic mistakes, while the others missed most issues—in some cases even failing to flag blatant and critical problems. It highlights a bigger problem with traditional SAST beyond just false positives: critical vulnerabilities quietly slipping into the codebase undetected. Through our research we are uncovering more and more details about just how poorly legacy SAST platforms perform.

Below we compare each tool’s accuracy on the VTM Django app, with code samples from the PR and analysis of which tools caught or missed each issue. (Spoiler: one well-known tool even declared the vulnerable code all clear with “no issues found”!) Let’s dive in.

Introduction to VTM (Vulnerable Task Manager)

VTM is an intentionally vulnerable Python/Django web application designed for security training and benchmarking. Similar to RailsGoat (for Rails), VTM includes common web app vulnerabilities for demonstration purposes. In our test, we introduced a Pull Request with several serious flaws in the Django code – including remote code execution, SQL injection, weak cryptography, and broken access control. These represent real-world bug categories that often slip past traditional static analysis:

• Logic Flaws & Broken Access Controls: Mistakes in authorization logic (e.g. letting the wrong users access data)
• IDOR Issues: Insecure direct object references allowing attackers to enumerate or access others’ data

Each example highlights why the issue matters in a Django context, and how the tools performed.

1. Remote Code Execution via Unsafe Deserialization

What Happened in the Pull Request: In the Django view code, a new utility was introduced to deserialize data using Python’s pickle module:

def deserialize_data(data: bytes):
   return pickle.loads(data)

This code directly unpickles data, which is dangerous. If an attacker can supply a malicious pickle byte stream, it can execute arbitrary code upon loading. Python pickles are not secure for untrusted data – they can instantiate or execute objects during loading. In a web app, this could lead to a remote code execution (RCE) vulnerability (Understanding OWASP 2017 A8 Insecure Deserialization - Information Security Stack Exchange). Django doesn’t use pickle for deserializing cookies by default, so this is an obvious red flag in the custom code.

Vulnerability Explanation (OWASP/Django Context): Untrusted deserialization is so dangerous that it was highlighted in OWASP’s Top 10 (2017) as Insecure Deserialization. Such flaws often result in RCE if an attacker can manipulate the serialized input. In Django (or any Python app), using pickle.loads on user input is equivalent to running whatever code an attacker packaged in that data. Safe alternatives include using JSON for data interchange or Django’s signing functions for trusted data, to avoid executing malicious payloads.

Tool Detection:

• DryRun Security: Found it (✓) – DryRun’s analysis recognized the use of pickle.loads as a critical issue. In fact, DryRun’s PR summary explicitly flagged “Remote Code Execution via pickle.loads() in the new code (New functionality django by cktricky · Pull Request #5 · DryRunSecuritySandbox/vtm · GitHub). DryRun’s contextual engine understood this logic flaw (unsafe deserialization) without needing a hard-coded rule.
• Semgrep: Found it (✓) – Semgrep has a rule for dangerous deserialization. It caught the pickle usage and even left a detailed comment in the PR: “Avoid using pickle, which is known to lead to code execution vulnerabilities. When unpickling, the data could be manipulated to run arbitrary code. Instead, consider serializing data as JSON or a similar format.” (New functionality django by cktricky · Pull Request #5 · DryRunSecuritySandbox/vtm · GitHub)
• . This shows Semgrep identified the pattern and provided guidance.
• SonarQube: Missed it (✗) – SonarQube missed this issue. 
• Snyk Code: Missed it (✗) – Surprisingly, Snyk did not report this issue (or any issue) at all. 
• CodeQL: Missed it (✗) – CodeQL missed this issue.

2. Code Injection via eval() Usage

What Happened in the Pull Request: Another function introduced was executing Python expressions dynamically:

def execute_command(command: str):
   return eval(command)

Here, using Python’s eval() on the command string means an attacker controlling that string can execute arbitrary Python code on the server. This is a straightforward code injection vulnerability. In a Django app, if command came from an HTTP request (directly or indirectly), an attacker could send input like __import__("os").system("rm -rf /") and eval would obediently run it. This is equivalent to remote code execution as well.

Vulnerability Explanation: The function eval() will evaluate a string as code. It’s dangerous by design if the string isn’t strictly controlled. OWASP’s general advice for code injection is to avoid such constructs entirely, since they blur the line between data and code. In Python, the mantra is often "never eval() input"—it’s akin to the infamous eval in JavaScript (also considered unsafe). This kind of vulnerability can allow an attacker to run system commands or take over the app server.

Tool Detection:

• DryRun Security: Found it (✓) – DryRun’s analysis caught this immediately. In the DryRun summary, it listed “Arbitrary code execution risk using eval()” as a separate issue introduced by the PR (New functionality django by cktricky · Pull Request #5 · DryRunSecuritySandbox/vtm · GitHub). DryRun’s contextual engine doesn’t rely on just pattern matching; it knows that using eval() in this manner is almost always a critical flaw in a Django application.
• Semgrep: Found it (✓) – Semgrep flagged the eval usage as well. The PR comment from Semgrep noted: “Detected the use of eval(). eval() can be dangerous if used to evaluate dynamic content. If this content can be input from outside, this may be a code injection vulnerability. Ensure evaluated content is not definable by external sources.” (New functionality django by cktricky · Pull Request #5 · DryRunSecuritySandbox/vtm · GitHub). Semgrep correctly identified the risky function and warned the developer.
• SonarQube: Missed it (✗) – SonarQube did not detect the use of eval as a security problem.
• Snyk Code: Missed it (✗) – Again, Snyk did not report this glaring issue. No errors or warnings about eval were present in the PR checks (New functionality django by cktricky · Pull Request #5 · DryRunSecuritySandbox/vtm · GitHub). This is surprising because eval() is a well-known dangerous function. Snyk’s miss here underscores a pattern: it did not catch any of the issues we planted in this Django PR. Whether due to a missing rule or something else, this was a critical miss and would be a significant risk to an organization’s code base.
• CodeQL: Missed it (✗) – CodeQL did not detect the issue which was incredibly surprising. We first suspected CodeQL’s logic is such that it wants to be certain user input can reach this function before marking it as insecure but it did flag our encryption routine as insecure without validating its invocation or usage so this indicates to us they lack robust enough rules for Python/Django.

3. SQL Injection in a Raw Database Query

What Happened in the Pull Request: The code added a function to fetch a user by username using a raw SQL query constructed with string concatenation:

def get_user_by_username(db, username: str):
   query = f"SELECT * FROM auth_user WHERE username = '{username}'"
   db.execute(query)

This is a classic SQL Injection scenario. The query directly interpolates the username input into SQL without any sanitization or parameterization. An attacker could input something like username = "'; DROP TABLE auth_user; --", and the resulting SQL would terminate the query and drop the users table. Django’s ORM normally prevents this by using parameterized queries, but here the developer bypassed the ORM and built a raw query unsafely.

Vulnerability Explanation (OWASP/Django Context): SQL Injection is one of the most dangerous web vulnerabilities – it has consistently been at or near #1 in OWASP’s Top 10 for years (Preventing SQL Injection Attacks With Python – Real Python). It allows attackers to execute arbitrary SQL commands on the database. In Django, if you use high-level QuerySet methods (e.g. User.objects.get(username=...)), the framework handles proper escaping/parameterization. But using raw SQL (via cursor.execute or Model.objects.raw()) with unsanitized input will reintroduce the risk. This bug could lead to data leakage or destruction of data (exfiltrating user info, dropping tables, etc.), which is catastrophic in an application.

Tool Detection:

• DryRun Security: Found it (✓) – DryRun identified the raw query vulnerability. The DryRun summary called out “SQL injection vulnerability in get_user_by_username() through raw SQL query” (New functionality django by cktricky · Pull Request #5 · DryRunSecuritySandbox/vtm · GitHub). This indicates DryRun’s engine understood the data flow: user input flows into a raw query string. It’s impressive because detection requires context (knowing username is likely user-controlled and used in a query). DryRun’s analyzer for SQLi triggered as expected, showing accuracy in catching this logic.
• Semgrep: Found it (✓) – Semgrep also flagged this. A PR comment from Semgrep warned about string concatenation in SQL: “Untrusted input concatenated with raw SQL query can result in SQL Injection. Use prepared statements instead. (In SQLAlchemy, use text() with parameters, etc.)” (New functionality django by cktricky · Pull Request #5 · DryRunSecuritySandbox/vtm · GitHub). Semgrep’s pattern-based rules detected the f-string usage in an SQL context. It provided remediation advice to use parameterized queries.
• SonarQube: Found it (✓) – SonarQube likely caught this as a definite vulnerability (not just a hotspot). Sonar has rules for detecting SQL injection in Python when it sees string construction for SQL commands. This issue would contribute to the “D Security Rating” on new code reported by Sonar (New functionality django by cktricky · Pull Request #5 · DryRunSecuritySandbox/vtm · GitHub). Out-of-the-box, SonarQube raises an issue whenever cursor.execute() is called with a non-parameterized query. So Sonar would have listed this as a security bug to fix.
• Snyk Code: Missed it (✗) – Consistent with its pattern in this test, Snyk did not flag the raw query vulnerability. No SQL injection warning was raised in the Snyk report (which had zero findings overall). Given SQLi is often straightforward for static analysis, this miss is notable. It suggests Snyk’s engine might require certain patterns or was overly reliant on seeing a known sink function; regardless, in this case it provided a false sense of security by reporting nothing. At this point in our testing, we questioned whether or not Snyk has support for Python & Django at all. Their documentation seems to suggest they do have support but require you to package your application in a specific way. Perhaps this is the reason they are unable to find security issues in pull requests.
• CodeQL: Missed it (✗) – At this point in our testing we began to become concerned about those using CodeQL for Python / Django projects. CodeQL is often touted as one of the most sophisticated legacy SAST tools yet misses a trivial and obvious SQL Injection pattern as well as several other glaring security flaws.

4. Weak Encryption (AES ECB Mode)

What Happened in the Pull Request: The code introduced a function to encrypt data using AES in Electronic Codebook (ECB) mode:

from Crypto.Cipher import AES‍

def encrypt_data_ecb(key, data):
   cipher = AES.new(key, AES.MODE_ECB)
   return cipher.encrypt(data)

This isn’t an immediately exploitable bug like the above, but it is a cryptographic weakness. AES in ECB mode is considered insecure because it doesn’t randomize each block – identical plaintext blocks produce identical ciphertext blocks. This can reveal patterns in the data and undermine confidentiality. In practice, ECB mode can leak information about encrypted data (the famous “ECB Penguin” example shows how an image encrypted with ECB still shows the outline of the original). In a Django app, if this were used to encrypt sensitive tokens or passwords, it would be a token-related security gap – the encryption might not actually protect the data as expected.

‍Vulnerability Explanation: ECB mode lacks an initialization vector and is not semantically secure. Modern cryptographic guidance (OWASP, NIST) strongly recommends using AES in CBC mode with a random IV or using an authenticated encryption mode like GCM. Using ECB is considered a design flaw; it might not allow an immediate hack on its own, but it significantly weakens security. If an attacker captures some ECB-encrypted tokens, they could compare ciphertext patterns to guess content or use known-plaintext attacks. It also indicates the developer’s security design is outdated, which could correlate with other mistakes.

Tool Detection:

• DryRun Security: Found it (✓) – DryRun’s summary noted “Weak AES encryption using ECB mode in encrypt_data_ecb()” (New functionality django by cktricky · Pull Request #5 · DryRunSecuritySandbox/vtm · GitHub). This shows DryRun is not only catching obvious vulns but also cryptographic issues that require understanding context (that ECB is being used). DryRun likely flagged this under a category of insecure cryptography. It’s the kind of finding many linters or SAST tools might label as a weakness or informational, but DryRun highlights it since it can lead to serious token/security issues if not addressed.
• Semgrep: Missed it (✗) – In our test run, Semgrep did not generate a warning for the use of ECB mode. It’s possible there wasn’t a rule enabled for this specific pattern (Semgrep’s ruleset is customizable, and an appropriate rule exists in the community, but maybe it wasn’t in the default set we used). So Semgrep caught no problem with this function, meaning a potential weakness in token encryption would slip by.
• SonarQube: Missed it (✗)  – SonarQube missed this finding entirely. It is likely their tool lacks a defined rule for cases such as this.
• Snyk Code: Missed it (✗) – Snyk did not report this either (consistent with its across-the-board miss). If Snyk provided no output, it obviously didn’t flag the insecure encryption. This indicates that certain categories like cryptographic best practices might not be covered or prioritized by its analysis. In a real scenario, that means a subtle but important issue like weak encryption could persist in code if relying solely on Snyk.
• CodeQL: Found it (✓) – CodeQL did not discover any flaws other than this one which we found to be interesting. The tool will tend not to flag security issues unless it feels it can prove these functions are executable and with user-supplied input. Unfortunately, this means dangerous code which may not yet be in use can enter into an application’s code base leaving significant gaps. At a minimum, a warning would make sense here.

5. Broken Access Control / Logic Flaw in Authentication

What Happened in the Pull Request: Perhaps the most insidious change was a function meant to check if a user can access the admin dashboard, which was implemented incorrectly:

def can_access_dashboard(user, account):
   if user.is_admin == False: # BUG: non-admin users also always get True
      return True
   else:
      return False   

In this flawed code, the intention was likely to allow access only for admins or staff, but due to a logic mistake, it returns True only for non-admins. This is a broken access control vulnerability. It essentially creates an authentication bypass via a logic flaw: any authenticated user could access the admin dashboard (or whatever sensitive functionality this check was protecting). This kind of bug is often called an improper authorization or authorization logic flaw because a user can do something they should not be allowed to do.

In a Django context, this might mean an endpoint meant for privileged users doesn’t actually enforce the restrictions. Attackers can enumerate through IDs or endpoints and reach data they aren’t supposed to see, because the check always passes. This is a pure logic flaw – no regex or taint rule will easily catch that the conditional statement is inverted and wrong. This is the exact type of issue that application security professionals have acknowledged is a limitation in legacy SAST tooling yet DryRun Security has proven its approach can improvise, it can go outside of the boundaries of looking only for specific rules and analyze the code for what it is.

‍Vulnerability Explanation (OWASP/Django Context): Broken access control is the #1 issue in OWASP Top 10 (2021) for good reason – 94% of applications have some form of access control weakness according to OWASP research. An IDOR or missing authorization check allows attackers to access or modify other users’ data by simply changing a parameter like a user ID or account number. In Django, developers must explicitly enforce permissions (via decorators like @login_required, object-level checks, or packages like django-guardian). If they make a logic error, the framework won’t magically catch it. The result can be an authentication bypass or privilege escalation. These flaws are highly critical but also among the hardest for traditional tools to detect, because they don’t involve suspicious API calls or patterns – it’s the absence or incorrect implementation of a check that’s the problem.

Tool Detection:

• DryRun Security: Found it (✓) – DryRun was able to identify this logic bug. In the DryRun summary it was noted as “Incorrect access control logic in can_access_dashboard() potentially exposing unauthorized access” (New functionality django by cktricky · Pull Request #5 · DryRunSecuritySandbox/vtm · GitHub). This is significant: DryRun understood the business logic intent and spotted the flaw (the always-True condition). Traditional static analyzers usually struggle here, because there’s nothing syntactically wrong – only the context reveals that this is a bug. DryRun’s contextual analysis deduced the intention of the dashboard access control and why the access control logic was likely flawed, flagging it as a security finding. This demonstrates the power of DryRun’s approach in catching logic flaws that other tools miss.
• Semgrep: Missed it (✗) – Semgrep did not catch this. It had no rule to detect a faulty auth condition. Unless a specific pattern (like a known vulnerable function) is present, Semgrep won’t know the intent. Here, the code looks legitimate to a simple static scan – returning True/False in a function. The security issue arises from the logic context (always returning True). Semgrep, being pattern-based, can’t easily flag such business logic mistakes without a tailor-made rule (which would be hard to generalize).
• SonarQube: Missed it (✗) – SonarQube also did not flag this authorization bypass. Sonar doesn’t have insight into the developer’s intent; it doesn’t realize that the conditional statement defeats the purpose of the authorization check. No dataflow or known vulnerability pattern is violated – it’s purely a semantic/logic error. This is a classic example of what application security calls a “logical flaw”, which typically requires human intellect and insights to find. SonarQube left this unnoticed, meaning a critical authorization bug could sail through code review if one relied only on Sonar.
• Snyk Code: Missed it (✗) – As with all other issues in this PR, Snyk didn’t report anything here. This isn’t surprising, they are a pattern-based tool and even if all conditions were met for them to successfully run, which would be a negative experience for developers in the first place, it would not be able to detect this type of issue. Again, the problem lies in the fundamental way these legacy SAST tools work.
• GitHub CodeQL: Missed it (✗) – CodeQL is a very powerful traditional SAST tool but therein lies the issue. It is a very powerful rule based scanner. It is impossible to create every permutation of rule which would be required to find authorization logic flaws in real applications. Again, an underlying limitation of this older technology and approach.

The Final Scoreboard

Analysis of each tool and how it performed in detecting our five security flaws we planted in the Django VTM application:

{{table5}}

As shown above, DryRun Security caught 5/5 vulnerabilities. In stark contrast, Snyk Code caught 0/5, flagging nothing at all—it completely missed every issue in this PR. We do wish to note that Snyk’s documentation mentions using a specific tool and pattern to essentially compile your code base for Python/Django so perhaps they are not well equipped for scanning these types of Pull Requests “out of the box”. SonarQube did slightly better and it caught 1/5. Semgrep managed 3/5, identifying the obvious patterns (pickle, SQL injection, eval) but missing the rest. GitHub CodeQL caught 1/5, only the weak ECB crypto usage, and missed all other high-impact problems.

These results underscore that traditional SAST solutions are often blind to critical vulnerabilities if they don’t match known patterns or rules. In this test, two of the tools (including a commercial one) would have given a green light to merge entirely vulnerable code—a frightening outcome for any security team.

It’s clear that when it comes to Python/Django security, especially logic flaws and context-specific bugs, the legacy tools left major gaps. SonarQube in particular provided a false sense of security by failing to flag even the critical issues, and Snyk’s “No issues found” message would have developers celebrating a dangerously insecure build. Semgrep and CodeQL fared slightly better on pattern-based findings, but fell short on anything requiring deeper understanding of the code’s intent or behavior.

Why These Misses Matter

The issues we introduced are not contrived edge cases; they represent real weaknesses that appear in Python/Django applications:

• Broken access control (IDOR) was the #1 OWASP Top 10 issue in 2021 for prevalence and impact. If your scanning tool can’t catch an auth bypass like our can_access_dashboard flaw, you could deploy code that completely undermines your app’s security model. Traditional SAST struggled here, whereas DryRun’s AI-based analysis understood the context and caught it.
• Logic flaws and IDOR issues (like an oversight that allows a user to enumerate through resources they shouldn’t see) often require understanding the app’s business logic. DryRun’s approach shines by using an LLM to reason about the code’s intent. The others, being pattern-based, largely miss such flaws because they aren’t described by a simple regex or known CVE.
• Cryptographic-related security gaps such as hardcoded secrets, poor token generation, or weak encryption (our AES ECB example) may not trigger high-severity alerts in traditional tools. Yet, these can be backdoors for attackers or can weaken your overall security posture (e.g., an API token encrypted with ECB could be deciphered if intercepted). DryRun treated the weak crypto as a notable finding, whereas all other tools missed it with the exception of CodeQL.
• Classic SQL Injection vulnerabilities were caught by most tools except Snyk and GitHub CodeQL. This is a reminder that no single tool is infallible; even well-known scanners can have blind spots. In our test, relying solely on one tool would have been dangerous. DryRun Security not only caught them, but also consolidated them into an easy-to-read PR summary with all findings listed in context (PR #5), making it very clear to the developer that the PR introduced serious issues.

Final Thoughts

For security professionals and AppSec engineers, the takeaway is clear: accuracy and context matter as much as coverage. In a Python/Django codebase, where you might have custom authentication logic or framework-specific patterns, a tool that understands the framework and logic can catch vulnerabilities that others miss. DryRun Security acted like an expert Django security reviewer—it spotted the mistakes that could lead to a compromise, and did so directly in the pull request in near real-time. 

Ready to See the Difference?

DryRun Security found every vulnerability and clearly communicated them. The other solutions, while sometimes catching low-hanging fruit, missed key threats—the kind that attackers won’t miss.

More head-to-head comparisons are on the way as we continue to test DryRun Security against other languages and frameworks in real-world scenarios. But you don’t have to wait for the next blog post to see these benefits for yourself. Reach out to us for a personalized DryRun Security demo.