Skip to content

Commit da9520b

Browse files
Kbayeroelmilan06ylladaAlexSanchez-bitJocLRojas
authored
Release/v11.2.6 (#2025)
* feat: add compliance control and query configuration tables with related DTOs and mappers * feat: add compliance control and query configuration tables with related DTOs and mappers * feat: add compliance control and query configuration tables with related DTOs and mappers * feat: add compliance control and query configuration tables with related DTOs and mappers * feat: add compliance control and query configuration tables with related DTOs and mappers * feat: add compliance control and query configuration tables with related DTOs and mappers * feat: add compliance control and query configuration tables with related DTOs and mappers * feat: add compliance control and query configuration tables with related DTOs and mappers * feat: Add new ControlConfig creation component for SQL‑based compliance evaluation * feat: Add new ControlConfig creation component for SQL‑based compliance evaluation # Conflicts: # frontend/src/environments/environment.ts * feat: Add new ControlConfig creation component for SQL‑based compliance evaluation * feat: Add new ControlConfig creation component for SQL‑based compliance evaluation * feat: Add new ControlConfig creation component for SQL‑based compliance evaluation * feat: Add new ControlConfig creation component for SQL‑based compliance evaluation * feat: Add new ControlConfig creation component for SQL‑based compliance evaluation * feat: Add new ControlConfig creation component for SQL‑based compliance evaluation * feat: add compliance control and query configuration tables with related DTOs and mappers * feat: add compliance control and query configuration tables with related DTOs and mappers * feat: add compliance control and query configuration tables with related DTOs and mappers * feat: add compliance control and query configuration tables with related DTOs and mappers * feat: enhance compliance control configuration with section mapping and DTO updates * feat: Revert unnecessary changes * feat: enhance compliance control configuration with section mapping and DTO updates * feat: refactor compliance query form and related components for improved layout and usability * feat: implement compliance orchestrator backend client and evaluation logic * feat: implement compliance orchestrator backend client and evaluation logic * feat: implement compliance orchestrator backend client and evaluation logic * feat: provide endpoint for OpenSearch evaluations including latest evaluation calculation per control * feat: implement compliance orchestrator backend client and evaluation logic * feat: provide endpoint for OpenSearch evaluations including latest evaluation calculation per control * feat: refactor compliance evaluation classes and update related mappings * feat: implement compliance orchestrator backend client and evaluation logic * feat: refactor compliance evaluation classes and update related mappings * feat: implement timeline visualization for compliance evaluations with initial chart setup and styling * feat: implement timeline visualization for compliance evaluations with initial chart setup and styling * feat: implement timeline visualization for compliance evaluations * feat: implement timeline visualization for compliance evaluations * feat: implement timeline visualization for compliance evaluations * feat: implement timeline visualization for compliance evaluations * feat: implement timeline visualization for compliance evaluations * feat: enrich compliance evaluation details with rule, rule value, and hits * feat: enhance compliance control evaluation with pagination support * feat: extend control evaluation status with additional states and refine evaluation logic * feat: add evaluation rule and rule value to compliance evaluation DTO and service * feat: include evaluation rule and rule value in query evaluation model and logic * feat: enhance compliance control evaluation with search functionality * feat: update application version file path for consistency * fix: implement sorting logic for compliance evaluations * feat(agent): add native auditd collector for Linux - Implement native auditd collector using go-libaudit v2 with netlink multicast - Add enterprise-ready auditd configuration (50-utmstack.rules) - Respect existing customer audit rules (additive approach) - Add cleanup on agent uninstall (removes UTMStack rules only) - Support automatic auditd installation on Debian/Ubuntu/RHEL/Fedora - Handle migration path for existing auditd installations - Add distro detection for package manager selection - Remove legacy beats/filebeat commented code * feat(filters): add auditd support to linux filter v5.0.0 - Support native auditd collector JSON format (type: auditd) - Map auditd fields to Standard Event Schema: - syscall/category -> action - result -> actionResult - exe/comm -> origin.process - proctitle -> origin.command - subj_user -> origin.user - cwd -> origin.path - exit -> statusCode (cast to int) - Set default severity 'info' for auditd events - Preserve numeric IDs in log.* for correlation rules - Maintain backwards compatibility with journald format * refactor(filters): remove deprecated system_linux_module.yml * fix(filters): adjust auditd event severity handling in linux filter * chore(agent): update version to 11.1.5 * changeset[backend](linux): update linux filter * fix(agent): prevent auditd buffer overflow with backpressure mitigation * fix(agent): reduce auditd log noise with threshold and execve filter - Add 50 event threshold for EventsLost logging (ignore 1-2 event losses) - Filter execve rules to real users only (auid>=1000, auid!=-1) - Simplify EventsLost function * fix(agent): filter false events lost from go-libaudit sequence rollover * feat(agent): expand auditd rules with log tampering and identity files * feat[backend](agent): added shell parameter to agent connection * feat[frontend](agent-console): added shell switch for windows agents (powershell or cmd) * fix[backend](elastic-service): added space verification before removing elastic-index * fix[backend](index-removal): added index verification before removal * fix[backend](index-removal): fixed index state field obtention * fix[backend](index-removal): fixed compilation errors on index removal * changeset[backend](o365_visualization): updated o365 file upload visualization * changeset[backend](o365_visualization): updated o365 file sync downloaded visualization * changeset[backend](o365_visualization): updated o365 visualizations * update windows-events filter * feat[backend](dependencies): updated apache-tika to avoid vulnerable version * feat[backend](dependencies): updated flying-saucer-pdf dependency and removed unneeded itext dependency * fix(installer): enhance post-installation error handling and Docker shutdown for security risks * changeset[backend](windows_filter): updated windows filters * changeset[backend](windows_rules): updated windows rules * feat[frotend](dependencies): updated dependencies for security improves * changeset[backend](windows_rules): updated windows rules data types * fix(installer): security improvements and code cleanup Security fixes: - Use crypto/rand instead of math/rand for secret generation - SELinux set to permissive instead of disabled (RedHat) - PostgreSQL/OpenSearch ports never exposed (use docker exec) - Nginx uses ephemeral key instead of INTERNAL_KEY Code improvements: - Remove unused parameters (GetAdminEmail, ConfigureNginx) - Remove dead code (if true condition) - Fix typo "fisnished" -> "finished" - Simplify PostInstallation (no Docker restart needed) - Remove unused dependencies (lib/pq, grequests) * feat(security): add OpenSearch SSL and authentication support * feat[backend](updated filters and rules): added a initial process to update logtash filters an rules * feat: add endpoint to retrieve latest evaluation by control ID * feat: enhance compliance evaluation mappers to load full Standard object * feat(soc-ai): add multi-provider LLM support and HTTP API for manual analysis - Add support for multiple LLM providers (OpenAI, Anthropic) with URL-based detection - Implement generic authentication via customHeaders configuration - Add HTTP API server on port 8090 for manual alert submission: - POST /api/v1/analyze - Submit alert for async LLM analysis - GET /api/v1/metrics - API request metrics - GET /health - Health check (unauthenticated) - Add X-Internal-Key authentication middleware for protected endpoints - Add AutoAnalyze config flag to enable/disable automatic processing - Add AnthropicRequest/Response schema types for Claude API format - Add ANTHROPIC_API_VERSION constant for required header - Clean up unused constants (GPT_API_ENDPOINT, AllowedGPTModels) - Fix silent JSON parsing errors with proper logging * feat(backend): add filters and rules to backend docker image * fix(backend): update OpenSearch connection to use HTTPS with authentication * feat(panel): add manual alert analysis endpoint with SSL support * fix(backend): use analyzeAlert method in UtmAlertServiceImpl * feat[backend](updated filters and rules): added initial load service * feat[backend](updated filters and rules): added initial load service * feat[backend](updated filters and rules): forced systemOwnedMode on rules and filters insertion in initial update * feat[backend](updated filters and rules): added removed rules and filters routines * feat[backend](updated filters and rules): forced filters adn rules to follow id convention of system owned rules/filters * feat[backend](updated filters and rules): set null to invalid module name rules * changeset[backend](socai): updated socai integration guide configuration * fix[frontend](integration-guide): fixed cisco asa and firepower commands * fix[backend](cypherUtil): make key|iv derivation be local instead of static * fix: correct query parameter for search * fix: Corrected incorrect behavior in filtering * fix: update sorting direction for compliance evaluations * feat: add print view for compliance evaluations with detailed report * feat: add print view for compliance evaluations with detailed report * feat: add print view for compliance evaluations with detailed report * feat: add print view for compliance evaluations with detailed report * feat: add print view for compliance evaluations with detailed report * feat: add print view for compliance evaluations with detailed report * feat: add print view for compliance evaluations with detailed report * feat: add print view for compliance evaluations with detailed report * fix[backend](healthcheck): removed springboot healthcheck to allow custom opensearch client to be used * fix[backend](initial-setup): improved filter insertion on system load * fix[backend](initial-setup): improved filter deletion on initial setup * changeset[backend](data-types,modules): seeded data type - module relation * fix[backend](initial-setup): improved error handling on failed to insert filter * fix[backend](initial-setup): fixed filter index error * fix[backend](initial-setup):added RuleYml to parse rules yml * fix[backend](rules): fixed rules definitions * feat[backend](log_events): added logs on every system admin operation * fix[rules]: remove unused rules * fix[baclend]: solve sintax problems in rules * fix[backend]: solved problem with merge * fix[backend](serialization): add @JsonIgnore to UtmModule lazy collections to prevent HttpMessageNotWritableException * fix[agent-manager](security): prevent SQL injection in filters, add constant-time key comparison, fix cache race condition, and add command timeout * feat[backend,frontend](incident-response): add shell selection for Windows agents, fix flow agent loading, enforce alert name in triggers, and rename default to dedicated agent * fix[frontend](soar): remove legacy automation route and update audit link to use flow editor * refactor(plugins): replace config polling with channel-based configuration updates across cloud integration plugins (AWS, Azure, GCP, O365, Sophos) * feat[soc-ai]: improve soc-ai integration * fix[backend,frontend,plugins](integrations): mask sensitive config values, improve validation error messages per provider, and prevent double-encryption * fix[frontend](integrations): only clear saved tenant changes instead of all pending changes * fix(modules-config): prevent single plugin failure from blocking all sync * fix[backend,frontend]: add missing Constants import and remove duplicate variable declaration * feat: refactor OpenSearch integration with new connection model * feat: rename compliance configuration changelogs * fix: correct standardId binding in compliance control create component * fix[frontend](alert-selection): improved selection/remove alert condition on table * fix[modules-config](socai): fixed providers configuration * fix[frontend](socai_module_config): fixed saving state handling * feat[modules-config](socai): generalized socai connection check and validations * feat: trigger initial scheduler dispatch when backend configs are present * fix[frontend](alert-selection): improved selection/remove alert condition on table * fix[modules-config](socai): fixed providers configuration * fix[frontend](socai_module_config): fixed saving state handling * feat[modules-config](socai): generalized socai connection check and validations * fix[frontend](socai_alert_analyze): updated loading status inmediately after request is maded * refactor(modules-config): improve resilience and code organization Add resilient module synchronization: - Implement periodic retry mechanism (5-minute interval) - Add StartPeriodicRetry for automatic recovery * feat: rename compliance configuration changelogs * fix[modules-config](socai): trimmend config values to avoid false positive on config verification * fix[modules-config](socai): send model on test request to avoid wrong model false positive * fix[modules-config](socai): add little message on model test to manage wrong models and wronk api keys on the request * fix[modules-config](socai): manage gemini test response correctly * fix[frontend](tooltips): moved tooltip position to body so they can be showed above any other copmonent * fix[frontend](alert-popup): fixed alert popup position * fix[backend](socai_model): removed model invalidation on custom provider * fix[frontend](socai-analysis): fixed wait status on socai alert analysis --------- Co-authored-by: Elena Lopez Milan <elopez@utmstack.com> Co-authored-by: Yadian Llada Lopez <yadian.llada@gmail.com> Co-authored-by: AlexSanchez-bit <sanchez.saez.alex01@gmail.com> Co-authored-by: JocLRojas <joc.l.rojas02@gmail.com> Co-authored-by: Osmany Montero <osmontero@icloud.com> Co-authored-by: Alex Sánchez <alex.sanchez@utmstack.com>
1 parent eab295c commit da9520b

File tree

1,027 files changed

+53869
-5730
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

1,027 files changed

+53869
-5730
lines changed

.github/workflows/reusable-java.yml

Lines changed: 12 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,11 @@ on:
3636
type: string
3737
default: "clean install"
3838
description: "Maven goals to execute"
39+
copy_filters_and_rules:
40+
required: false
41+
type: boolean
42+
default: false
43+
description: "Copy filters and rules folders to build context"
3944

4045
jobs:
4146
build:
@@ -99,6 +104,13 @@ jobs:
99104
username: utmstack
100105
password: ${{ secrets.GITHUB_TOKEN }}
101106

107+
- name: Copy filters and rules to build context
108+
if: ${{ inputs.copy_filters_and_rules }}
109+
run: |
110+
cp -r filters ./${{ inputs.image_name }}/
111+
cp -r rules ./${{ inputs.image_name }}/
112+
echo "✅ Copied filters and rules to ./${{ inputs.image_name }}/"
113+
102114
- name: Build and Push the Image
103115
uses: docker/build-push-action@v6
104116
with:

.github/workflows/v11-deployment-pipeline.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -493,6 +493,7 @@ jobs:
493493
use_tag_as_version: true
494494
maven_profile: 'prod'
495495
maven_goals: 'clean package'
496+
copy_filters_and_rules: true
496497

497498
build_frontend:
498499
name: Build Frontend Microservice

agent-manager/agent/agent_imp.go

Lines changed: 36 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -29,13 +29,20 @@ type AgentService struct {
2929
AgentStreamMap map[uint]AgentService_AgentStreamServer
3030
AgentStreamMutex sync.Mutex
3131
CacheAgentKey map[uint]string
32-
CacheAgentKeyMutex sync.Mutex
32+
CacheAgentKeyMutex sync.RWMutex
3333
CommandResultChannel map[string]chan *CommandResult
3434
CommandResultChannelM sync.Mutex
3535

3636
DBConnection *database.DB
3737
}
3838

39+
func (s *AgentService) ValidateAgentKey(key string, id uint) bool {
40+
s.CacheAgentKeyMutex.RLock()
41+
defer s.CacheAgentKeyMutex.RUnlock()
42+
_, valid := utils.IsKeyPairValid(key, id, s.CacheAgentKey)
43+
return valid
44+
}
45+
3946
func InitAgentService() error {
4047
var err error
4148
agentServOnce.Do(func() {
@@ -338,20 +345,35 @@ func (s *AgentService) ProcessCommand(stream PanelService_ProcessCommandServer)
338345
return status.Errorf(codes.Internal, "failed to send command to agent: %v", err)
339346
}
340347

341-
result := <-s.CommandResultChannel[cmdID]
342-
err = s.DBConnection.Upsert(
343-
&models.AgentCommand{},
344-
"agent_id = ? AND cmd_id = ?",
345-
map[string]interface{}{"command_status": models.Executed, "result": result.Result},
346-
cmd.AgentId, cmdID,
347-
)
348-
if err != nil {
349-
catcher.Error("failed to update command status", err, map[string]any{"process": "agent-manager"})
350-
}
348+
select {
349+
case result := <-s.CommandResultChannel[cmdID]:
350+
err = s.DBConnection.Upsert(
351+
&models.AgentCommand{},
352+
"agent_id = ? AND cmd_id = ?",
353+
map[string]interface{}{"command_status": models.Executed, "result": result.Result},
354+
cmd.AgentId, cmdID,
355+
)
356+
if err != nil {
357+
catcher.Error("failed to update command status", err, map[string]any{"process": "agent-manager"})
358+
}
351359

352-
err = stream.Send(result)
353-
if err != nil {
354-
return err
360+
err = stream.Send(result)
361+
if err != nil {
362+
return err
363+
}
364+
case <-time.After(5 * time.Minute):
365+
s.CommandResultChannelM.Lock()
366+
delete(s.CommandResultChannel, cmdID)
367+
s.CommandResultChannelM.Unlock()
368+
369+
_ = s.DBConnection.Upsert(
370+
&models.AgentCommand{},
371+
"agent_id = ? AND cmd_id = ?",
372+
map[string]interface{}{"command_status": models.Error, "result": "command timed out after 5 minutes"},
373+
cmd.AgentId, cmdID,
374+
)
375+
376+
return status.Errorf(codes.DeadlineExceeded, "agent did not respond within 5 minutes")
355377
}
356378

357379
s.CommandResultChannelM.Lock()

agent-manager/agent/collector_imp.go

Lines changed: 8 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -43,13 +43,20 @@ type CollectorService struct {
4343
CollectorConfigsCache map[uint][]*CollectorConfigGroup
4444
CollectorConfigsCacheM sync.Mutex
4545
CacheCollectorKey map[uint]string
46-
CacheCollectorKeyMutex sync.Mutex
46+
CacheCollectorKeyMutex sync.RWMutex
4747
CollectorPendigConfigChan chan *CollectorConfig
4848
CollectorTypes []enum.UTMModule
4949

5050
DBConnection *database.DB
5151
}
5252

53+
func (s *CollectorService) ValidateCollectorKey(key string, id uint) bool {
54+
s.CacheCollectorKeyMutex.RLock()
55+
defer s.CacheCollectorKeyMutex.RUnlock()
56+
_, valid := utils.IsKeyPairValid(key, id, s.CacheCollectorKey)
57+
return valid
58+
}
59+
5360
func InitCollectorService() {
5461
collectorServOnce.Do(func() {
5562
CollectorServ = &CollectorService{

agent-manager/agent/interceptor.go

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -2,7 +2,7 @@ package agent
22

33
import (
44
"context"
5-
_ "errors"
5+
"crypto/subtle"
66
"fmt"
77
"strconv"
88
"strings"
@@ -79,11 +79,11 @@ func authHeaders(md metadata.MD, fullMethod string) error {
7979
typ := strings.ToLower(connectorType[0])
8080
switch typ {
8181
case "agent":
82-
if _, isValid := utils.IsKeyPairValid(key, uint(id), AgentServ.CacheAgentKey); !isValid {
82+
if !AgentServ.ValidateAgentKey(key, uint(id)) {
8383
return status.Error(codes.PermissionDenied, "invalid key")
8484
}
8585
case "collector":
86-
if _, isValid := utils.IsKeyPairValid(key, uint(id), CollectorServ.CacheCollectorKey); !isValid {
86+
if !CollectorServ.ValidateCollectorKey(key, uint(id)) {
8787
return status.Error(codes.PermissionDenied, "invalid key")
8888
}
8989
default:
@@ -102,7 +102,7 @@ func authHeaders(md metadata.MD, fullMethod string) error {
102102
}
103103

104104
func isInternalKeyValid(token string) bool {
105-
return token == config.InternalKey
105+
return subtle.ConstantTimeCompare([]byte(token), []byte(config.InternalKey)) == 1
106106
}
107107

108108
func isInRoute(route string, list []string) bool {

agent-manager/agent/parser.go

Lines changed: 5 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -100,7 +100,11 @@ func replaceSecretValues(input string) string {
100100
return match
101101
}
102102
encryptedValue := matches[2]
103-
decryptedValue, _ := utils.DecryptValue(config.EncryptionKey, encryptedValue)
103+
decryptedValue, err := utils.DecryptValue(config.EncryptionKey, encryptedValue)
104+
if err != nil {
105+
catcher.Error("failed to decrypt secret value in command", err, map[string]any{"process": "agent-manager"})
106+
return match
107+
}
104108
return decryptedValue
105109
})
106110
}

agent-manager/utils/auth.go

Lines changed: 7 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,7 @@
11
package utils
22

33
import (
4+
"crypto/subtle"
45
"crypto/tls"
56
"net/http"
67
"strings"
@@ -19,10 +20,12 @@ func IsConnectionKeyValid(panelUrl string, token string) bool {
1920
}
2021

2122
func IsKeyPairValid(key string, id uint, cache map[uint]string) (string, bool) {
22-
for agentId, agentKey := range cache {
23-
if key == agentKey && id == agentId {
24-
return agentKey, true
25-
}
23+
agentKey, ok := cache[id]
24+
if !ok {
25+
return "", false
26+
}
27+
if subtle.ConstantTimeCompare([]byte(key), []byte(agentKey)) == 1 {
28+
return agentKey, true
2629
}
2730
return "", false
2831
}

agent-manager/utils/filter.go

Lines changed: 27 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,7 @@ package utils
22

33
import (
44
"fmt"
5+
"regexp"
56
"strings"
67

78
"gorm.io/gorm"
@@ -24,14 +25,14 @@ type Filter struct {
2425
Value interface{}
2526
}
2627

27-
func NewFilter(searchQuery string) []Filter {
28-
defer func() {
29-
if r := recover(); r != nil {
30-
// Handle the panic here
31-
fmt.Println("Panic occurred:", r)
32-
}
33-
}()
28+
// validFieldName ensures the field name only contains safe characters (letters, digits, underscores)
29+
var validFieldName = regexp.MustCompile(`^[a-zA-Z_][a-zA-Z0-9_]*$`)
3430

31+
func IsValidFieldName(field string) bool {
32+
return validFieldName.MatchString(field)
33+
}
34+
35+
func NewFilter(searchQuery string) []Filter {
3536
filters := make([]Filter, 0)
3637
if searchQuery == "" {
3738
return filters
@@ -42,11 +43,27 @@ func NewFilter(searchQuery string) []Filter {
4243
}
4344
for _, v := range query {
4445
filter := strings.Split(v, "=")
46+
if len(filter) != 2 {
47+
continue
48+
}
4549
filerQuery := strings.Split(filter[0], ".")
50+
if len(filerQuery) != 2 {
51+
continue
52+
}
53+
field := filerQuery[0]
54+
if !IsValidFieldName(field) {
55+
fmt.Printf("Rejected invalid filter field: %s\n", field)
56+
continue
57+
}
58+
op := resolveOperator(filerQuery[1])
59+
if op == "" {
60+
continue
61+
}
4662
filters = append(filters, Filter{
47-
Field: filerQuery[0],
48-
Op: resolveOperator(filerQuery[1]),
49-
Value: filter[1]})
63+
Field: field,
64+
Op: op,
65+
Value: filter[1],
66+
})
5067
}
5168
return filters
5269
}

agent-manager/utils/paginator.go

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,7 +28,19 @@ func NewPaginator(limit int, page int, sort string) Pagination {
2828
if len(sort) > 0 {
2929
srt := make([]string, 0)
3030
for _, s := range strings.Split(sort, "&") {
31-
srt = append(srt, strings.Replace(s, ",", " ", 1))
31+
parts := strings.SplitN(s, ",", 2)
32+
field := parts[0]
33+
if !IsValidFieldName(field) {
34+
continue
35+
}
36+
direction := "asc"
37+
if len(parts) == 2 {
38+
d := strings.ToLower(strings.TrimSpace(parts[1]))
39+
if d == "desc" {
40+
direction = "desc"
41+
}
42+
}
43+
srt = append(srt, field+" "+direction)
3244
}
3345
p.Sort = strings.Join(srt, ",")
3446
}

agent/cmd/uninstall.go

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -47,6 +47,15 @@ var uninstallCmd = &cobra.Command{
4747
if err = pb.DeleteAgent(cnf); err != nil {
4848
utils.Logger.ErrorF("error deleting agent: %v", err)
4949
}
50+
51+
// Uninstall dependencies (cleanup auditd rules, etc.)
52+
fmt.Print("Cleaning up dependencies... ")
53+
if err = dependency.UninstallAll(); err != nil {
54+
fmt.Printf("Warning: %v\n", err)
55+
} else {
56+
fmt.Println("[OK]")
57+
}
58+
5059
if err = collector.UninstallAll(); err != nil {
5160
fmt.Printf("error uninstalling collectors: %v\n", err)
5261
os.Exit(1)

0 commit comments

Comments
 (0)