Seamless Jenkins-Keycloak Integration: Overcoming API Authentication Challenges

Problem

The client faced an issue integrating Jenkins with Keycloak for authentication. While the Jenkins UI successfully authenticated users via Keycloak, API calls from backend services were failing. According to Jenkins’ documentation, API requests should be authenticated using an API token, but despite following the recommended steps, the client encountered authentication failures (403 Forbidden & 401 Unauthorized errors) when calling Jenkins REST endpoints. Specifically, the errors included:

• 403 No valid crumb included in the request – indicating Jenkins’ CSRF protection was blocking the request.
• 401 Unauthorized – suggesting API authentication was failing.
• Authentication redirects (commenceLogin) – implying Jenkins was treating requests as unauthenticated.

Process

Step 1: Initial Investigation

The expert first analyzed the Jenkins authentication logs and Keycloak token requests to determine the cause of the failure. Key observations included:

• The API token was not being recognized properly in CSRF-protected requests.
• Jenkins was treating API requests as coming from an anonymous user without necessary permissions.
• The Keycloak access token was not being validated correctly by Jenkins.

To troubleshoot, the expert suggested the following approaches:

• Check if CSRF protection was causing the issue by attempting to disable it.
• Retrieve a crumb manually and include it in API requests.
• Test authentication using different token formats (Bearer token, API token, and Personal Access Token).
• Verify Keycloak-Jenkins integration settings to ensure token validation was correctly configured.
• Check Jenkins user permissions to confirm the API user had the necessary roles.

Step 2: Troubleshooting & Fixes

CSRF Protection Adjustments

The client attempted to disable CSRF protection but found no option in the UI. The expert recommended modifying Jenkins’ configuration file (jenkins.xml) to disable CSRF protection manually. However, due to security concerns, the client preferred keeping CSRF protection enabled.

Including a Crumb in API Requests

The expert suggested retrieving a crumb before making API calls:

curl -k -X GET "https:///crumbIssuer/api/json" -H "Authorization: Bearer "

The client reported failures retrieving the crumb, receiving 401 Unauthorized errors.

Using API Token Instead of Access Token

The expert suggested using an API token instead of a Keycloak bearer token for authentication:

curl -k -X GET "https:///crumbIssuer/api/json" --user "test_admin:"

However, this also resulted in a 401 Unauthorized error.

Checking Jenkins User Permissions

The expert advised verifying that the test_admin user had proper permissions under Manage Jenkins → Manage Users → Assign Roles. The client confirmed that Overall/Administer permission was granted but the issue persisted.

Anonymous Read Access & Authentication Handling

The client observed that enabling anonymous access allowed API authentication to work, but disabling it caused failures. The expert suggested explicitly allowing authenticated users to access the crumbIssuer API while keeping other endpoints protected.

Using Cookie-Based Authentication

To bypass authentication redirects, the expert recommended a cookie-based login approach:

curl -c cookies.txt -u "test_admin:" "https:///j_acegi_security_check"

curl -b cookies.txt -k -X GET "https:///crumbIssuer/api/json"

This approach ensured that authentication was preserved across multiple API requests.

Checking Keycloak Integration

The expert advised verifying Keycloak’s client settings, ensuring the client_id, client_secret, and realm settings matched Jenkins’ configuration. The client attempted using client credentials grant instead of user-based authentication but still faced access issues.

Solution

After extensive testing, the client successfully resolved the issue by:

• Registering an API token in the Jenkins user settings.
• Using the API token to retrieve a crumb before making API requests.
• Ensuring API requests included the correct crumb and authentication token.
• Confirming that Jenkins recognized the authenticated API user with proper permissions.

The final working API request looked like this:

curl -k -X GET "https:///crumbIssuer/api/json" --user "test_admin:"

curl -v -k -X POST --user "test_admin:" -H "Jenkins-Crumb: " "https:///job//buildWithParameters" --data process_id=123

By following this structured approach, the client achieved seamless authentication for API calls while maintaining security best practices.

Conclusion

The issue stemmed from Jenkins not correctly recognizing authentication tokens from Keycloak and CSRF protection blocking API requests. The expert’s step-by-step guidance helped the client:

• Identify authentication failures and permission issues.
• Implement a reliable API token-based authentication mechanism.
• Ensure secure and stable API access without compromising Jenkins’ security settings.

Ultimately, the structured troubleshooting approach led to a successful integration of Jenkins with Keycloak while maintaining both authentication security and API accessibility.

​