Skip to content

Add condor_jobview to show logfile even without indexer. Add api to s… #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
mmascher wants to merge 1 commit into
base: master
Choose a base branch
from
Open

Add condor_jobview to show logfile even without indexer. Add api to s… #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions glideinmonitor/indexer/indexer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@ def directory_jobs(start_path):

# Iterate through each folder in the starting directory (probably GWMS_Log_Dir)
for root, dirs, files in os.walk(start_path):
log("DEBUG", "Indexing directory. Data root %s dirs %s" % (str(root), str(dirs)))

# Child directory level
level = root.replace(start_path, '').count(os.sep)

Expand Down
2 changes: 1 addition & 1 deletion glideinmonitor/lib/database.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -253,7 +253,7 @@ def getInfo(self, jobID, given_guid):

# Do directory/file names need to be sanitized?
if given_guid:
cur.execute("SELECT * FROM file_index WHERE GUID='{}'".format(jobID))
cur.execute("SELECT * FROM file_index WHERE GUID LIKE '{}'".format(jobID))

@mambelli mambelli Sep 14, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do you expect partial GUIDs?
Why otherwise use a LIKE that is generally less performance than a comparison?

@mmascher mmascher Sep 24, 2021

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, tt could be useful to access logs from condor job id for example. Although now that I implemented the new condor_jobview API this is not needed by CMS anymore.

else:
cur.execute("SELECT * FROM file_index WHERE ID='{}'".format(jobID))

Expand Down
2 changes: 1 addition & 1 deletion glideinmonitor/lib/logger.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,4 +24,4 @@ def log(error_level, message):
os.makedirs(log_location_dir)
log_location = os.path.join(log_location_dir, datetime.datetime.now().strftime("%Y-%m-%d") + ".txt")
with open(log_location, "a") as log_file:
log_file.write(error_level + " - " + str(int(datetime.datetime.now().timestamp())) + " - " + message + "\n")
log_file.write(error_level + " - " + "{:%Y-%m-%d %H:%M:%S%z}".format(datetime.datetime.now()) + " - " + message + "\n")

@mambelli mambelli Sep 14, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

datetime.now is by default naive (no time zone info). To print full iso8601 w/ timezone use:
datetime.datetime.now().astimezone().isoformat() or not to have microseconds datetime.datetime.now().astimezone().replace(microsecond=0).isoformat() and to have a single format string:

log_file.write(f"{error_level} - {datetime.datetime.now().astimezone().replace(microsecond=0).isoformat()} - {message}\n")

6 changes: 6 additions & 0 deletions glideinmonitor/webserver/rest_api.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,12 @@ def api_job_file(job_id, given_guid):
return path


def api_condorjob_file(factory, feuser, entry_name, condor_job_id):
# Get configuration

return "/data/glideinmonitor/upload/%s/user_%s/glidein_gfactory_instance/entry_%s/job.%s" % (factory, feuser, entry_name, condor_job_id)

@mambelli mambelli Sep 14, 2021

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be better to use a configuration variable for the root directory. Is this similar to GWMS_Log_Dir except the client subdir?



def api_job_info(job_id, given_guid):
# Provides info on a job in a JSON format
db = Database()
Expand Down
168 changes: 168 additions & 0 deletions glideinmonitor/webserver/static/condor_jobview.html
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,168 @@
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8">
<title>GWMS Factory - Condor Job View</title>

<link href="/assets/libs/bootstrap.min.css" rel="stylesheet"
integrity="sha384-ggOyR0iXCbMQv3Xipma34MD+dH/1fQ784/j6cY/iJTQUOhcWr7x9JvoRxT2MZw1T" crossorigin="anonymous">

<link href="https://cdn.jsdelivr.net/gh/gitbrent/bootstrap4-toggle@3.5.0/css/bootstrap4-toggle.min.css"
rel="stylesheet">

<style>
.container {
padding-top: 50px;
}

.code tr th {
width: 10px;
}
</style>
</head>
<body>
<div id="app">
<nav class="navbar navbar-expand-lg navbar-dark bg-dark">
<div class="container" style="padding-top: 10px; padding-bottom: 10px">
<a class="navbar-brand" href="/">GWMS Factory Job Logs</a>
<button class="navbar-toggler" type="button" data-toggle="collapse" data-target="#mainNavbarCollapse"
aria-controls="mainNavbarCollapse" aria-expanded="false" aria-label="Toggle navigation">
<span class="navbar-toggler-icon"></span>
</button>

<div class="collapse navbar-collapse" id="mainNavbarCollapse">
<ul class="navbar-nav mr-auto">
<li class="nav-item active">
<a class="nav-link" href="/">Home <span class="sr-only">(current)</span></a>
</li>
</ul>
</div>
</div>
</nav>
<div class="container">

<div class="card">
<div class="card-header"><h3>
Job
<strong>{{job_id}}</strong>
</h3>
</div>
<div class="card-body">
<div class="row">
<div class="col-sm">
<h4>General Information</h4>
<table class="table table-hover">
<tbody>
<tr>
<th scope="row">FileSize <small>(.err + .out)</small></th>
<td>{{infoFileSize}}</td>
</tr>
<tr>
<th scope="row">Entry Name</th>
<td>{{infoEntryName}}</td>
</tr>
<tr>
<th scope="row">Frontend Username</th>
<td>{{infoFrontendUsername}}</td>
</tr>
</tbody>
</table>
</div>
<div class="col-sm">
<div class="row">
<h4>Data Files </h4>
</div>
<div class="row">
<div class="col-sm">
<input id="toggleFileAction" type="checkbox" checked data-toggle="toggle" data-on="Open"
data-off="Download" data-onstyle="secondary" data-offstyle="secondary">
</div>
</div>
<div style="padding-top: 10px" class="row">
<div class="col-sm">
<div style="max-width: 300px">
<h5>Full Logs</h5>
<button id="btnJobOUT" disabled type="button"
class="btn btn-outline-secondary btn-lg btn-block text-justify">
{{job_id}}.out
<ion-icon name="arrow-forward"></ion-icon>
</button>
<br>
<button id="btnJobERR" disabled type="button"
class="btn btn-outline-secondary btn-lg btn-block text-justify">
{{job_id}}.err
<ion-icon name="arrow-forward"></ion-icon>
</button>
<br>
<button id="btnJobGZIP" type="button"
class="btn btn-outline-primary btn-lg btn-block text-justify">
{{job_id}}.tar.gz
<ion-icon name="arrow-forward"></ion-icon>
</button>
</div>
</div>
<div class="col-sm">
<div style="max-width: 300px">
<h5>Condor Logs</h5>
<button id="btnMasterLog" disabled type="button"
class="btn btn-outline-secondary btn-lg btn-block text-justify">Master Log
<ion-icon name="arrow-forward"></ion-icon>
</button>
<br>
<button id="btnStartdLog" disabled type="button"
class="btn btn-outline-secondary btn-lg btn-block text-justify">Startd Log
<ion-icon name="arrow-forward"></ion-icon>
</button>
<br>
<button id="btnStarterLog" disabled type="button"
class="btn btn-outline-secondary btn-lg btn-block text-justify">Starter Log
<ion-icon name="arrow-forward"></ion-icon>
</button>
<br>
<button id="btnStardHistLog" disabled type="button"
class="btn btn-outline-secondary btn-lg btn-block text-justify">StardHist
Log
<ion-icon name="arrow-forward"></ion-icon>
</button>
<br>
<button id="btnXMLDescLog" disabled type="button"
class="btn btn-outline-secondary btn-lg btn-block text-justify">XML
Description
<ion-icon name="arrow-forward"></ion-icon>
</button>
<br>
</div>
</div>
</div>
</div>
</div>
</div>
</div>

<!-- Totally efficient way to allow someone to scroll past the bottom of the screen, I am a good programmer -->
<br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br><br>
</div>

<script src="/assets/libs/jquery.min.js"
integrity="sha256-CSXorXvZcTkaix6Yvo6HppcZGetbYMGWSFlBw8HfCJo="
crossorigin="anonymous"></script>
<script src="/assets/libs/bootstrap.min.js"
crossorigin="anonymous"></script>
<script src="/assets/libs/vue.min.js"
integrity="sha256-chlNFSVx3TdcQ2Xlw7SvnbLAavAQLO0Y/LBiWX04viY="
crossorigin="anonymous"></script>
<script src="/assets/libs/moment.min.js"
crossorigin="anonymous"></script>
<script type="module" src="https://unpkg.com/ionicons@4.5.10-0/dist/ionicons/ionicons.esm.js"></script>
<script nomodule="" src="https://unpkg.com/ionicons@4.5.10-0/dist/ionicons/ionicons.js"></script>

<script src="/assets/libs/FileSaver.min.js" crossorigin="anonymous"></script>

<script src="/assets/libs/bootstrap4-toggle.min.js"></script>


<script src="/assets/bundle.js"></script>
<script src="/assets/condor_jobview.js"></script>
</div>
</body>
</html>
Loading