We go head-to-head, comparing DryRun Security to Snyk Code, GitHub Advanced Security (CodeQL), Semgrep, and SonarQube in a Ruby on Rails application.
Today’s static application security testing (SAST) tools vary widely in how accurately they catch vulnerabilities and how usable they are for developers. We decided it’d be useful to do a head-to-head comparison using some intentionally vulnerable code in RailsGoat—specifically changes introduced in this PR #9 in our forked repo. This repo had DryRun Security running alongside Snyk Code, GitHub Advanced Security (CodeQL), Semgrep, and SonarQube/SonarCloud.
We focused on accuracy of findings, the ability to catch non-pattern-based issues, and overall coverage (including logic flaws, authorization mishaps, and SSRF). Below is a detailed comparison and a head-to-head look at why DryRun Security delivers superior SAST accuracy and effectiveness for a modern application security program. Note that here we are using DryRun Security out of the box and chose not to use Natural Language Code Policies (NLCP) for this head-to-head comparison.
The Short Comparison Grid
Here’s a concise table comparing each tool’s accuracy, coverage, scanning speed, false positives, and developer experience. We tested each tool on the changes made in PR #9, which contained a mix of obvious vulnerabilities (like SQL Injection and XSS) plus less obvious logic flaws (like user enumeration, authorization bypasses, and SSRF).
{{table}}
Evaluating the Tools: Who Gets It Right?
To ground our comparison in code, we tested several major vulnerability categories in a pull request: RailsGoat PR #9. This PR modifies or adds code in ways that demonstrate:
- SQL Injection (SQLi)
- Cross-Site Scripting (XSS)
- Server-Side Request Forgery (SSRF)
- Logic/Authorization Flaws (e.g., IDOR or missing auth checks)
- User Enumeration (detailed error messages)
- Hardcoded / Insecure Token Generation
Below are short summaries of each category and how each tool measured up. We finish with a scoreboard table showing final tallies.
1. SQL Injection (SQLi)
What Happened in the Pull Request (DryRunSecuritySandbox/railsgoat - PR #9):
def username_lookup
user = ActiveRecord::Base.connection.execute("SELECT * FROM users WHERE username = '#{params[:username]}'")
...
end
Additionally, another raw query in a model method uses string interpolation without parameterization.
Vulnerability Explanation (OWASP / Rails Context):
SQL Injection allows attackers to alter queries by injecting malicious SQL commands. Although Rails’ ActiveRecord typically parameterizes queries, developers can bypass these safeguards by manually concatenating user input. OWASP’s SQL Injection Prevention Cheat Sheet discusses the severity of SQLi, while the Rails Security Guide details how to safely handle user input.
Tool Detection
- Snyk: Found the injection in the controller (✓), but missed the second query in the model (partial).
- GitHub Advanced Security (CodeQL): Found the main injection (✓), also missing the second one (partial).
- Semgrep: Missed it (✗).*
- SonarQube: Missed it (✗).
- DryRun Security: Found both instances (✓).
2. Cross-Site Scripting (XSS)
What Happened in the Pull Request (DryRunSecuritySandbox/railsgoat - PR #9):
<p>
Here is your content: <%= params[:user_input].html_safe %>
</p>
Vulnerability Explanation (OWASP / Rails Context):
XSS occurs when attackers inject scripts into pages viewed by other users. Rails automatically escapes variables by default, but using html_safe, raw, or <%== %> can disable this. OWASP’s XSS Prevention Cheat Sheet explains how reflected, stored, or DOM-based XSS can be introduced. Rails docs caution developers against using html_safe on untrusted inputs (see Rails Security Guide on XSS).
Tool Detection
- Snyk: Found it (✓).
- GitHub Advanced Security (CodeQL): Found it (✓).
- Semgrep: Found it (✓).
- SonarQube: Missed it (✗).
- DryRun Security: Found it (✓).
3. Server-Side Request Forgery (SSRF)
What Happened in the Pull Request (DryRunSecuritySandbox/railsgoat - PR #9):
def client_request
target_url = params[:external_resource]
uri = URI.parse(target_url)
response = Net::HTTP.get(uri)
end
Vulnerability Explanation (OWASP / Rails Context):
SSRF flaws arise when the server fetches a remote resource using a URL supplied by the user, allowing attackers to potentially reach internal services or cloud metadata endpoints. OWASP’s SSRF Prevention Cheat Sheet describes how these attacks pivot into private networks or read sensitive data. In Rails, classes like Net::HTTP or OpenURI can be misused for SSRF if user input isn’t validated.
Tool Detection
- Snyk: Missed it (✗).
- GitHub Advanced Security (CodeQL): Missed it (✗).
- Semgrep: Found it (✓).
- SonarQube: Missed it (✗).
- DryRun Security: Found it (✓).
4. Logic/Authorization Flaws (IDOR / Missing Auth Checks)
What Happened in the Pull Request (DryRunSecuritySandbox/railsgoat - PR #9):
def show_confidential_info
data = ConfidentialModel.find(params[:id])
render json: data
end
Vulnerability Explanation (OWASP / Rails Context):
Insecure direct object references (IDOR) let users access data by manipulating an identifier (e.g., /record/123). If no authorization check exists, an attacker can read or modify resources they don’t own. OWASP’s Broken Access Control references IDOR as a top vulnerability. Rails doesn’t automatically enforce authorization for resource-based actions—developers must implement checks (e.g., using Pundit, CanCanCan, or manual logic) to prevent unauthorized data access.
Tool Detection
- Snyk: Missed it (✗).
- GitHub Advanced Security (CodeQL): Missed it (✗).
- Semgrep: Missed it (✗).
- SonarQube: Missed it (✗).
- DryRun Security: Found it (✓).
5. User Enumeration (Detailed Error Messages)
What Happened in the Pull Request (DryRunSecuritySandbox/railsgoat - PR #9):
if user.nil?
render json: { error: "User not found: #{params[:username]}" }
else
unless user.authenticate(params[:password])
render json: { error: "Incorrect password for user #{params[:username]}" }
end
end
Vulnerability Explanation (OWASP / Rails Context):
By returning different responses when a user doesn’t exist vs. when a password is incorrect, attackers can brute-force valid usernames. This practice is covered by the OWASP Testing Guide on user enumeration and guessable user accounts. Rails developers often display friendlier messages, but it’s recommended to use a generic message like “Invalid username or password” for all login failures.
Tool Detection
- Snyk: Missed it (✗).
- GitHub Advanced Security (CodeQL): Missed it (✗).
- Semgrep: Missed it (✗).
- SonarQube: Missed it (✗).
- DryRun Security: Found it (✓).
6. Hardcoded / Insecure Token Generation
What Happened in the Pull Request (DryRunSecuritySandbox/railsgoat - PR #9):
def generate_jwt(user)
"fake-jwt-token-for-user-#{user.id}"
end
Vulnerability Explanation (OWASP / Rails Context):
Tokens must be unpredictable and securely generated. Hardcoding or predictable generation lets attackers guess tokens and impersonate other users. OWASP warns against this in various references to session management and token randomness. In Rails, the Security Guide notes that secret keys and session tokens must be random. Libraries like SecureRandom are typically used for robust token generation; never rely on incremental IDs or static strings.
Tool Detection
- Snyk: Missed it (✗).
- GitHub Advanced Security (CodeQL): Missed it (✗).
- Semgrep: Missed it (✗).
- SonarQube: Missed it (✗).
- DryRun Security: Found it (✓).
The Final Scoreboard
Here’s the scoreboard of who caught each vulnerability in our sample pull request (RailsGoat PR #9):
{{table2}}
In plain numbers:
- Snyk: 2/6 (partial SQLi, XSS)
- GitHub Advanced Security (CodeQL): 2/6 (partial SQLi, XSS)
- Semgrep: 2/6 (XSS, partial SSRF)*
- SonarQube: 0/6
- DryRun Security: 6/6
Why DryRun Security Wins
From these findings, DryRun Security clearly outperforms the competition. While Snyk, CodeQL, and Semgrep did okay on classic pattern-based issues (XSS, SSRF), they struggle with anything requiring deeper logic analysis, like authorization issues, user enumeration, or insecure tokens. SonarQube, in its default configuration, missed everything.
1. Contextual Security Analysis Goes Beyond Pattern Matching
DryRun Security digs into how your code processes data (e.g., user-supplied URLs, tokens, or error messages), instead of merely searching for known signatures. That’s why it caught SSRF in a Net::HTTP.get call and recognized that multiple error messages in an auth flow amount to user enumeration. Traditional SAST often can’t handle such nuanced logic.
2. Real-Time, Developer-Friendly Feedback
Instead of burying issues or issuing a bunch of GitHub comments, DryRun Security posts just one straightforward PR comment explaining what’s wrong, why it’s risky, and how to fix it. The feedback arrives quickly—often within seconds—keeping developers in their workflow and preventing critical vulnerabilities from hitting production.
3. Extendable Natural Lang Code Policies (NLCP)
While we didn’t use NLCP here, it’s worth emphasizing that DryRun Security supports Natural Language Code Policies, meaning you can craft new rules in plain English. AppSec teams love this because it doesn’t require specialized rule languages (like CodeQL or Semgrep patterns). For example, you could easily define, “Don’t allow token generation that doesn’t use SecureRandom” or “Reject any reference to internal IP ranges in user-supplied URLs” and “Does this change allow users to submit unrestricted GraphQL queries?”. DryRun then enforces those policies automatically in every PR.
4. Comprehensive Coverage with Minimal Noise
The scoreboard proves DryRun’s thoroughness. Yet it doesn’t flood teams with false positives. We know we didn’t cover false positives in the samples, but stay tuned for exciting future posts! By focusing on real vulnerabilities, DryRun fosters trust in the tool’s alerts—developers know that when DryRun flags something, it’s genuinely worth investigating.
Bringing It All Together
DryRun Security detects serious security bugs that other tools consistently miss. DryRun Security combines automated intelligence, policy-driven scanning, and a developer-friendly interface makes it ideal for DevSecOps teams that can’t afford to let SSRF, IDOR, BOLA, or insecure tokens slip through. DryRun goes beyond mere pattern matching to catch hidden, context-driven vulnerabilities.
Pair that with:
- Instant alerts in GitHub or Slack,
- Human-readable policies (no specialized syntax),
- Support for new languages and frameworks on demand,
…and you have a SAST solution that not only catches known vulnerabilities but adapts to the ever-changing threat landscape. The riskiest flaws aren’t always the “classic” CVEs—often, they’re hidden in your application logic, waiting to be exploited.
With DryRun Security, it’s like having an AppSec expert reviewing every pull request—only faster, more accurately, and around the clock.
Ready to see the difference?
If you're tired of juggling multiple scanners, missing critical logic flaws, and drowning in false positives, see how DryRun Security helps you catch real risks that other SAST tools can’t find. Schedule a 30-minute demo today.
*Correction: March 18, 2025
An engineer from Semgrep let us know that the most recent version of the rule does catch SQLi although it did not at the time of our initial blog post publication.
