`. Feeling bold? Set up :doc:`background tasks <../advanced/backgroundtasks>`.
diff --git a/docs/customization/create_broker.md b/docs/customization/create_broker.md
deleted file mode 100644
index aa84fd4e0..000000000
--- a/docs/customization/create_broker.md
+++ /dev/null
@@ -1,223 +0,0 @@
-Creating an Alert Broker Module for the TOM Toolkit
----------------------------------------------------
-
-This guide will walk you through how to create a custom alert broker module using the TOM toolkit.
-
-At the end of this tutorial we will have a very simple module that connects to
-an "alert broker" (in this case a static json file) and allows us to ingest
-targets into our TOM.
-
-You can follow this example to build an alert broker module to connect to a real
-alert broker.
-
-Be sure you've followed the [Getting Started](/introduction/getting_started) guide before continuing onto this tutorial.
-
-### What is an Alert Broker Module?
-A TOM Toolkit Alert Broker Module is an object which contains the logic for querying a remote broker
-(e.g [MARS](https://mars.lco.global)), and transforming the returned data into TOM Toolkit Targets.
-
-#### TOM Alerts module
-The TOM Alerts module is a Django app which provides the methods and
-classes needed to create a custom TOM alert broker module. A module may be created to ingest
-alerts of an arbitrary form from a remote source. The TOM Alerts module provides
-tools to transform these alerts into TOM-specific alerts to be used in the creation of TOM Targets.
-
-### Project Structure
-After following the [Getting Started](/introduction/getting_started) guide, you will have
-a Django project directory of the form:
-
-```
-mytom
-├── db.sqlite3
-├── manage.py
-└── mytom
- ├── __init__.py
- ├── settings.py
- ├── urls.py
- └── wsgi.py
-```
-
-### Creating a Broker Module
-In this example, we will create a broker named __MyBroker__.
-
-Begin by creating a file `my_broker.py`, and placing it in the inner `mytom/` directory
-of the project (in the directory with settings.py). `my_broker.py` will contain the classes that define our custom
-TOM Alert Broker Module.
-
-Our custom broker module relies on the TOM Toolkit modules that were installed in the
-[Getting Started](/introduction/getting_started) guide. Begin by editing `my_broker.py`
-to import the necessary modules.
-
-```python
-from tom_alerts.alerts import GenericQueryForm, GenericAlert, GenericBroker
-from tom_alerts.models import BrokerQuery
-from tom_targets.models import Target
-```
-
-In order to add custom forms to our broker module, we will also need Django's `forms` module.
-
-```python
-from django import forms
-```
-See [Working with Django Forms](https://docs.djangoproject.com/en/2.1/topics/forms/)
-
-Finally, import `requests` so that we can fetch some remote broker test data.
-
-```python
-import requests
-```
-See [Requests Official API Docs](http://docs.python-requests.org/en/master/)
-
-#### Test Data
-
-In place of a remote broker, we've uploaded a [sample JSON file to GitHub Gist](https://gist.githubusercontent.com/mgdaily/f5dfb4047aaeb393bf1996f0823e1064/raw/5e6a6142ff77e7eb783892f1d1d01b13489032cc/example_broker_data.json).
-
-For our `my_broker.py` module to use this data, we will set `broker_url` to it.
-```
-broker_url = 'https://gist.githubusercontent.com/mgdaily/f5dfb4047aaeb393bf1996f0823e1064/raw/5e6a6142ff77e7eb783892f1d1d01b13489032cc/example_broker_data.json'
-```
-
-#### Broker Forms
-To define the query forms for our custom broker module, we'll begin by creating class
-`MyBrokerForm` inside `my_broker.py`, which inherits the `tom_alert` module's
-`GenericQueryForm`.
-
-This will define the list of forms to be presented within the broker query. For
-our example, we'll be querying simply on target name.
-
-```python
-class MyBrokerForm(GenericQueryForm):
- target_name = forms.CharField(required=True)
-```
-
-#### Broker Class
-To define our broker module, we'll create the class `MyBroker`, also inside of `my_broker.py`.
-Our broker class will encapsulate the logic for making queries to a remote alert broker,
-retrieving and sanitizing data, and creating TOM alerts from it.
-
-Begin by defining the class, its name and default form. In our case, the name
-will simply be 'MyBroker', and the form will be `MyBrokerForm` - the form that we
-just defined!
-
-```python
-class MyBroker(GenericBroker):
- name = 'MyBroker'
- form = MyBrokerForm
-```
-
-#### Required Broker Class Methods
-Each TOM alert broker module is required to have a base set of class methods. These
-methods enable the conversion of remote alert data into TOM-specific
-alerts and targets.
-
-##### `fetch_alerts` Class Method
-`fetch_alerts` is used to query the remote broker, and return an iterator
-of results depending on the parameters passed into the query, so that
-these results may be displayed on the query results page. In our case, `fetch_alerts`
-will only filter on name, but this can be easily extended to other query parameters.
-
-```python
-@classmethod
-def fetch_alerts(clazz, parameters):
- response = requests.get(broker_url)
- response.raise_for_status()
- test_alerts = response.json()
- return iter([alert for alert in test_alerts if alert['name'] == parameters['target_name']])
-```
-**Why an iterator?** Because some alert brokers work by sending streams, not fully
-evaluated lists. This simple example broker could easily return a list (in fact we
-are coercing the list into an iterator!) but that would not work in the model
-where a broker is sending an unending stream of alerts.
-
-Our implementation will get a response from our test broker source, check that our
-request was successful, and return a iterator of alerts whose name field matches the
-name passed into the query.
-
-##### `to_generic_alert` Class Method
-In order to standardize alerts and display them in a consistent manner,
-the `GenericAlert` class has been defined within the `tom_alerts` library.
-This broker method converts a remote alert into a TOM Toolkit `GenericAlert`.
-
-```python
-@classmethod
-def to_generic_alert(clazz, alert):
- return GenericAlert(
- timestamp=alert['timestamp'],
- url=broker_url,
- id=alert['id'],
- name=alert['name'],
- ra=alert['ra'],
- dec=alert['dec'],
- mag=alert['mag'],
- score=alert['score']
- )
-```
-In our case, the `GenericAlert` attributes match up *almost* directly with our test
-data. How convenient! We'll just go ahead and define the `GenericAlert`'s `url`
-field as the `broker_url` we retrieved our test data from.
-
-```python
-...
-url=broker_url,
-...
-```
-
-#### Other methods
-
-`fetch_alerts` and `to_generic_alert` are the only methods required for your
-broker module to function. Of course you are free to add any number of additional
-methods or attributes to the module that you deem necessary.
-
-### Using Our New Alert Broker
-Now that we've created our TOM alert broker, let's hook it into our TOM
-so that we can ingest alerts and create targets.
-
-The `tom_alerts` module will look in `settings.py` for a list of alert
-broker classes, so we'll need to add `MyBroker` to that list.
-
-```python
-TOM_ALERT_CLASSES = [
- ...
- 'tom_alerts.brokers.mars.MARSBroker',
- 'mytom.my_broker.MyBroker',
- ...
-]
-```
-Now, navigate to the top-level directory of your Django project,
-where `manage.py` resides and run
-
-```bash
-./manage.py makemigrations
-./manage.py migrate
-./manage.py runserver
-```
-
-Navigate to [http://127.0.0.1:8000/alerts/query/list/](http://127.0.0.1:8000/alerts/query/list/)
-
-You should now see 'MyBroker' listed as a broker! Clicking the link will bring you
-to the query page, where you can make a query to our sample dataset.
-
-
-
-#### Making a Query
-
-Since we're only going to be filtering on the alert's 'target_name' field, we're only
-presented with that option. Name the query whatever you'd like, and we'll check
-our remote data source for a target named 'Tatooine'
-
-
-
-Going back to [http://127.0.0.1:8000/alerts/query/list/](http://127.0.0.1:8000/alerts/query/list/),
-our new query will appear. Click the 'run' button to run the query.
-
-
-
-The query result will be presented.
-
-
-
-To create a target from any query result, click the 'create target' button. To view the raw
-alert data, click the 'view' link.
-
-[Click here](https://gist.github.com/mgdaily/19aefebd05da91fe6ebfe928b4862a51) to view
-the full source code detailed in this example.
diff --git a/docs/customization/customize_observations.md b/docs/customization/customize_observations.md
deleted file mode 100644
index 2c58ff0a0..000000000
--- a/docs/customization/customize_observations.md
+++ /dev/null
@@ -1,297 +0,0 @@
-Changing How Observations are Submitted
----------------------------------------
-
-The LCO Observation module for the TOM Toolkit ships with a default HTML form that
-facilitates submitting basic observations to the LCO network. It may sometimes be
-desirable to customize the form to show or hide fields, add new parameters, or
-change the submission logic itself, depending on the needs of the project. In this
-tutorial we will customize our LCO module to submit multiple observations with
-different filters at the same time.
-
-This guide assumes you have followed the [getting
-started](/introduction/getting_started) guide and have a working TOM up and running.
-
-### Create a new Observation Module
-
-Many methods of customizing the TOM Toolkit involve inheriting/extending existing
-functionality. This time will be no different: we'll crate a new observation
-module that inherits the existing functionality from
-`tom_observations.facilities.LCOFacility`.
-
-First, create a python file somewhere in your project to house your new module.
-For example it could live next to your `settings.py`, or if you've started a new
-app, it could live there. It doesn't really matter, as
-long as it's located somewhere in your project:
-
- touch mytom/mytom/lcomultifilter.py
-
-Now add some code to this file to create a new observation module:
-
-```python
-# lcomultifilter.py
-from tom_observations.facilities.lco import LCOFacility
-
-
-class LCOMultiFilterFacility(LCOFacility):
- name = 'LCOMultiFilter'
-```
-So what does the above code do?
-
-1. Line 1 imports the LCOFacility that is already shipped with the TOM Toolkit. We
-want this class because it contains functionality we will re-use in our own
-implementation.
-2. Line 4 defines a new class named `LCOMultiFilterFacility` that inherits from
-`LCOFacility`.
-3. Line 5 sets the name attribute of this class to `LCOMultiFilter`.
-
-What you have done is created a new observation module that is functionally
-identical to the existing LCO module, but has a different name: `LCOMultiFilter`.
-A good start!
-
-Now we need to tell our TOM where to find our new module so we can use it to
-submit observations. Add (or edit) the following lines to your `settings.py`:
-
-```python
-# settings.py
-TOM_FACILITY_CLASSES = [
- 'tom_observations.facilities.lco.LCOFacility',
- 'tom_observations.facilities.gemini.GEMFacility',
- 'mytom.lcomultifilter.LCOMultiFilterFacility',
-]
-```
-This code lists all of the observation modules that should be available to our
-TOM.
-
-With that done, go to any target in your TOM and you should see your new module in
-the list:
-
-
-
-You could now use the new module now to make an observation, and it would work the
-same as the old LCO module.
-
-Note that if you see an error like: "There was a problem authenticating with LCO"
-then you need to [add your LCO api key](/docs/customsettings#facilities) to your
-`settings.py` file.
-
-### Adding additional fields
-
-Now that you've created a new observation module that's functionally the same as
-the old LCO module, how do we change it? One thing that might be useful is to add some extra
-fields to the form: two more choices of filters and exposure times. Back in the
-`lcomultifilter.py` file add a new import and create a new class that will become
-the new form:
-
-```python
-# lcomultifilter.py
-from tom_observations.facilities.lco import LCOFacility, LCOObservationForm, filter_choices
-from django import forms
-
-
-class LCOMultiFilterForm(LCOObservationForm):
- filter2 = forms.ChoiceField(choices=filter_choices)
- exposure_time2 = forms.FloatField(min_value=0.1)
- filter3 = forms.ChoiceField(choices=filter_choices)
- exposure_time3 = forms.FloatField(min_value=0.1)
-
-
-class LCOMultiFilterFacility(LCOFacility):
- name = 'LCOMultiFilter'
- form = LCOMultiFilterForm
-```
-
-There is now a new class, `LCOMultiFilterForm` which inherits from
-`LCOObservationForm`, the form for the default interface. Additionally there are
-definitions for 4 fields: `fiter2`, `exposure_time2`, `filter3`, and
-`exposure_time3`.
-
-A `form` attribute has been added on the `LCOMultiFilterFacility`
-class, this tells our observation module to use the new `LCOMultiFilterForm`
-instead of the default LCO observation form.
-
-
-### Modifying the form layout
-
-Now that the desired fields have been added to the `LCOMultiFilterForm`, the
-form's layout needs to be modified in order to actually display them. In this
-example we'll split the form into two rows: one row for the three filter choices
-and exposure times, and another row for everything else. Note that the default
-form already has fields for `filter` and `exposure_time`, so we'll overwrite the
-entire layout so that they appear next to the new fields we added.
-
-The `LCOObservationForm` has a method `layout()` that returns the desired layout
-using the [crispy forms Layout](https://django-crispy-forms.readthedocs.io/en/d-0/layouts.html)
-class. Familiarizing yourself with the basic functionality of crispy forms would
-be a good idea if you wish to deeply customize your observation module's form.
-
-With our modified layout added, the `lcomultifilter.py` file now looks like this:
-
-```python
-# lcomultifilter.py
-from tom_observations.facilities.lco import LCOFacility, LCOObservationForm, filter_choices
-from django import forms
-from crispy_forms.layout import Div
-
-
-class LCOMultiFilterForm(LCOObservationForm):
- filter2 = forms.ChoiceField(choices=filter_choices)
- exposure_time2 = forms.FloatField(min_value=0.1)
- filter3 = forms.ChoiceField(choices=filter_choices)
- exposure_time3 = forms.FloatField(min_value=0.1)
-
- def layout(self):
- return Div(
- Div(
- Div(
- 'name', 'proposal', 'ipp_value', 'observation_type', 'start', 'end',
- css_class='col'
- ),
- Div(
- 'instrument_name', 'exposure_count', 'max_airmass',
- css_class='col'
- ),
- css_class='form-row'
- ),
- Div(
- Div(
- 'filter', 'exposure_time',
- css_class='col'
- ),
- Div(
- 'filter2', 'exposure_time2',
- css_class='col'
- ),
- Div(
- 'filter3', 'exposure_time3',
- css_class='col'
- ),
- css_class='form-row'
- )
- )
-
-
-class LCOMultiFilterFacility(LCOFacility):
- name = 'LCOMultiFilter'
- form = LCOMultiFilterForm
-```
-
-Take a look at the layout and compare it to the [existing lco layout](https://github.com/TOMToolkit/tom_base/blob/master/tom_observations/facilities/lco.py#L169). A second
-row has been added that includes all the filter choices. Note that the original
-`filter` and `exposure_time` have been moved from their original location to the
-new row.
-
-Now if you select "LCOMultiFilter" from the list of observation facilities on a
-target you should see your new form:
-
-
-
-Is the form still too ugly for you? Trying playing with the layout definition to
-suit your needs.
-
-### Changing the form submission behavior
-
-If you are not familiar with the [LCO submission
-API](https://developers.lco.global/#observations) now might be a good time to take
-a look. The LCO Observation module uses this API to submit observations using the
-data provided in the form, so we need to modify how this happens. More
-specifically, we'd like to add two additional `Configuration` to our observation
-request, one for each of our additional filters and exposure times.
-
-Using the `observation_payload()` method, we can use `super()` to get the
-original LCO module's observation request, then modify it to suit the needs of our
-`LCOMultiFilter` class:
-
-```python
-#lcomultifilter.py
-from tom_observations.facilities.lco import LCOFacility, LCOObservationForm, filter_choices
-from django import forms
-from crispy_forms.layout import Div
-from copy import deepcopy
-
-class LCOMultiFilterForm(LCOObservationForm):
- filter2 = forms.ChoiceField(choices=filter_choices)
- exposure_time2 = forms.FloatField(min_value=0.1)
- filter3 = forms.ChoiceField(choices=filter_choices)
- exposure_time3 = forms.FloatField(min_value=0.1)
-
- def layout(self):
- return Div(
- Div(
- Div(
- 'name', 'proposal', 'ipp_value', 'observation_type', 'start', 'end',
- css_class='col'
- ),
- Div(
- 'instrument_type', 'exposure_count', 'max_airmass',
- css_class='col'
- ),
- css_class='form-row'
- ),
- Div(
- Div(
- 'filter', 'exposure_time',
- css_class='col'
- ),
- Div(
- 'filter2', 'exposure_time2',
- css_class='col'
- ),
- Div(
- 'filter3', 'exposure_time3',
- css_class='col'
- ),
- css_class='form-row'
- )
- )
-
- def observation_payload(self):
- payload = super().observation_payload()
- configuration2 = deepcopy(payload['requests'][0]['configurations'][0])
- configuration3 = deepcopy(payload['requests'][0]['configurations'][0])
- configuration2['instrument_configs'][0]['optical_elements']['filter'] = self.cleaned_data['filter2']
- configuration2['instrument_configs'][0]['exposure_time'] = self.cleaned_data['exposure_time2']
- configuration3['instrument_configs'][0]['optical_elements']['filter'] = self.cleaned_data['filter3']
- configuration3['instrument_configs'][0]['exposure_time'] = self.cleaned_data['exposure_time3']
- payload['requests'][0]['configurations'].extend([configuration2, configuration3])
- return payload
-
-
-class LCOMultiFilterFacility(LCOFacility):
- name = 'LCOMultiFilter'
- form = LCOMultiFilterForm
-```
-
-Let's go over what we did in this new `observation_payload()` method:
-
-1. Line 1: We call `super().observation_payload()` to get the observation request
-which the parent class (LCOFacility) would have called.
-2. Line 2-3 We copy the Request's Configuration into two new Configurations: `configuration2` and
-`configuration3`. These will be the additional Configuration we send to LCO.
-3. Lines 5-8: We set the value of these new Configuration `filter` and
-`exposure_time` to the values we collected from our custom form.
-4. lines 10-11: Finally, we extend the original Request's Configuration array to
-include the 2 new Configuration we built. Return it and we're done!
-
-If you submit an observation request with the `LCOMultiFilter` observation module
-now you should see that it creates an observation request with LCO with three
-Configuration!
-
-### Summary
-
-Our original requirement was to be able to submit observations to LCO with some
-additional filters and exposure times. We accomplished this by:
-
-1. Creating a new observation module: a `LCOMultiFilterFacility` class and a
-`LCOMultiFilterForm`, both of which were child classes of the original
-`LCOFacility` class (since we wanted to keep most of the functionality intact) and
-then added this new class to our `TOM_FACILITY_CLASSES` setting.
-
-2. We added a few fields to `LCOMultiFilterForm` and modified it's layout to
-include these new fields using `layout()`.
-
-3. We implemented the `LCOMultiFilterForm` `observation_payload()` which used the
-parent's class return value and then modified it to suit our needs.
-
-This is a good example of Object Oriented Programming in Python. If you are
-curious about how this all works, we recommend reading up on OOP in general, as
-well as how objects in Python 3 work.
diff --git a/docs/customization/customize_template_tags.rst b/docs/customization/customize_template_tags.rst
new file mode 100644
index 000000000..38d80fe9c
--- /dev/null
+++ b/docs/customization/customize_template_tags.rst
@@ -0,0 +1,322 @@
+Customizing Template Tags
+=========================
+
+The TOM Toolkit is designed to be as customizable as possible. A number
+of UI objects are rendered as Django templatetags. Django has quite a
+few `built-in template
+tags `__,
+but also allows the creation of `custom template
+tags `__,
+which the TOM Toolkit leverages heavily.
+
+However, it’s possible that a TOM Toolkit template tag doesn’t quite
+meet your needs. Maybe the axis labels for photometry plotting aren’t
+quite what you’re looking for, or the target data isn’t formatted the
+way you’d like. This tutorial will show you how to write your own
+template tag to suit your own program better.
+
+Preparing your project for custom template tags
+-----------------------------------------------
+
+The first thing your project will need is a custom app. You can read
+about custom apps in the Django tutorial
+`here `__, but
+to quickly get started, the command to create a new app is as follows:
+
+.. code:: python
+
+ ./manage.py startapp custom_code
+
+Where ``custom_code`` is the name of your app. You will also need to
+ensure that ``custom_code`` is in your ``settings.py``. Append it to the
+end of ``INSTALLED_APPS``:
+
+.. code:: python
+
+ ...
+ INSTALLED_APPS = [
+ 'django.contrib.admin',
+ 'django.contrib.auth',
+ ...
+ 'tom_dataproducts',
+ 'custom_code',
+ ]
+ ...
+
+You should now have a directory within your TOM called ``custom_code``,
+which looks like this:
+
+::
+
+ ├── custom_code
+ | ├── __init__.py
+ │ ├── admin.py
+ │ ├── apps.py
+ │ ├── models.py
+ │ ├── tests.py
+ │ └── views.py
+ ├── data
+ ├── db.sqlite3
+ ├── manage.py
+ ├── mytom
+ │ ├── __init__.py
+ │ ├── settings.py
+ │ ├── urls.py
+ │ └── wsgi.py
+ ├── static
+ ├── templates
+ └── tmp
+
+Next, you’ll need to add a ``templatetags`` directory within
+``custom_code``. Create an empty file called ``__init__.py`` within that
+directory. Finally, we need a file to put the code for our custom
+template tags. Add a file in ``custom_code`` called ``custom_extras``.
+It’s convention to use ``_extras`` within your template tag module name.
+
+Your ``custom_code`` directory should look like this:
+
+::
+
+ └── custom_code
+ ├── __init__.py
+ ├── admin.py
+ ├── apps.py
+ ├── models.py
+ ├── templatetags
+ | ├── __init__.py
+ | └── custom_extras.py
+ ├── tests.py
+ └── views.py
+
+Writing a custom template tag
+-----------------------------
+
+For our template tag, we’re going to write a tag that displays the
+timestamp and magnitude for the most recent photometry point available
+for a target. There are three aspects to a template tag:
+
+- The code in ``custom_extras`` to run the logic to get the data we’ll
+ be displaying
+- The partial template to render the data
+- Putting the custom tag somewhere we’d like it displayed
+
+The Python code
+~~~~~~~~~~~~~~~
+
+We’re going to write a ``recent_photometry`` function in our
+``custom_extras`` first. Step one is the necessary import and
+initialization of the template library:
+
+.. code:: python
+
+ from django import template
+
+
+ register = template.Library()
+
+Now, to the ``recent_photometry`` function. A couple notes about the
+approach here:
+
+- The function will have the decorator ``@register.inclusion_tag()``.
+ There are a couple of different types of template tags, but we’re
+ using the ``inclusion_tag`` because it renders a template, allowing
+ us to customize how it looks. The ``simple_tag`` is a different type
+ of template tag that simply modifies data, so that won’t work for us.
+- Within the decorator is a path to the partial template that will
+ render the data–this doesn’t exist yet, but remember the file name
+ we’re using!
+- We’d like to get the latest photometry values for a specific target,
+ so we’ll need to pass a ``Target`` as a parameter.
+- We’d also like to be able to specify how many photometry points we
+ care about, so let’s also include a keyword argument that defaults to
+ just 1.
+
+.. code:: python
+
+ from django import template
+
+
+ register = template.Library()
+
+
+ @register.inclusion_tag('custom_code/partials/recent_photometry.html')
+ def recent_photometry(target, num_points=1):
+ return {}
+
+You can see that we’ll eventually be returning a dictionary, but first
+we need to add our logic. We’ll need to use the ``Target`` passed in to
+get all ``ReducedDatum`` objects for that ``Target`` with a
+``data_type`` of ``photometry``. Then we’ll need to order by
+``timestamp`` descending, and slice just the first few. Make sure to
+take note of the imports in this step!
+
+.. code:: python
+
+ import json
+
+ from django import template
+
+ from tom_dataproducts.models import ReducedDatum
+
+
+ register = template.Library()
+
+
+ @register.inclusion_tag('custom_code/partials/recent_photometry.html')
+ def recent_photometry(target, num_points=1):
+ photometry = ReducedDatum.objects.filter(data_type='photometry').order_by('-timestamp')[:num_points]
+ return {'recent_photometry': [(datum.timestamp, json.loads(datum.value)['magnitude']) for datum in photometry]}
+
+It’s only a couple of lines, but there’s a lot going on here. The first
+line does the aforemention database query and slices the first point of
+the ``QuerySet``. The second line constructs a dictionary–the only key
+is ``recent_photometry``, and the corresponding value is a list of
+tuples. Each tuple has the timestamp as the first item, and the
+magnitude as the second item.
+
+Ultimately, this template tag will, when included, return the most
+recent photometry points for a ``Target``. But it can’t display
+anything!
+
+The partial template
+~~~~~~~~~~~~~~~~~~~~
+
+So now we need to create
+``custom_code/templates/custom_code/partials/recent_photometry.html``.
+We’ll need to add yet another series of directories and files. Your
+directory structure should now look like this:
+
+Let’s start with the partial template. We’ll need to add yet another
+series of directories and files. Add the following to your directory
+structure:
+
+::
+
+ └── custom_code
+ └── templates
+ └── custom_code
+ └── partials
+ └── recent_photometry.html
+
+Your complete directory structure should look like this:
+
+::
+
+ └── custom_code
+ ├── __init__.py
+ ├── admin.py
+ ├── apps.py
+ ├── models.py
+ ├── templates
+ | └── custom_code
+ | └── partials
+ | └── recent_photometry.html
+ ├── templatetags
+ | ├── __init__.py
+ | └── custom_extras.py
+ ├── tests.py
+ └── views.py
+
+And let’s open up ``recent_photometry.html`` and get to work.
+
+.. code:: html
+
+
+
+
+ | Timestamp | Magnitude |
|---|
+
+ {% for datum in recent_photometry %}
+
+ | {{ datum.0 }} |
+ {{ datum.1 }} |
+
+ {% empty %}
+
+ | No recent photometry. |
+
+ {% endfor %}
+
+
+
`` element.
+
+So we have a partial template and a template tag that can be used
+anywhere, but we have to put it somewhere!
+
+Using the template tag
+~~~~~~~~~~~~~~~~~~~~~~
+
+The target detail page seems like a logical place for this, so let’s go
+there. First, we need to override our ``target_detail.html`` template.
+If you haven’t read the tutorial on template overriding, you can do so
+`here `__– in the meantime, you’ll need to add
+``target_detail.html`` to ``templates/tom_targets/`` in the top level of
+your project. Your project directory should look like this:
+
+::
+
+ ├── custom_code
+ ├── data
+ ├── db.sqlite3
+ ├── manage.py
+ ├── mytom
+ ├── static
+ ├── templates
+ │ └── tom_targets
+ │ └── target_detail.html
+ └── tmp
+
+Then, you’ll need to copy the contents of ``target_detail.html`` in the
+base TOM Toolkit to your ``target_detail.html``. You can find that file
+on
+`Github `__.
+
+Near the top of the file, there’s a series of template tags that are
+loaded in. Add ``custom_extras`` to that list:
+
+.. code:: html
+
+ {% extends 'tom_common/base.html' %}
+ {% load comments bootstrap4 tom_common_extras targets_extras observation_extras dataproduct_extras publication_extras custom_extras static cache %}
+ ...
+
+Then, put your templatetag in the HTML somewhere, passing in ``object``
+(which refers to the object value of the current template context) and
+the desired number of photometry points:
+
+.. code:: html
+
+ ...
+ {% endif %}
+ {% target_buttons object %}
+ {% target_data object %}
+ {% if object.type == 'SIDEREAL' %}
+ {% aladin object %}
+ {% endif %}
+ {% recent_photometry object num_points=3 %}
+ ...
+
+The new table should be displayed on your target detail page! Not only
+that, but you’ll now be able to include that template tag on other
+pages, too. And if it doesn’t quite meet your needs–perhaps you want the
+most recent photometry points for all targets, for example–it can be
+easily modified.
+
+As far as this template tag goes, as of this tutorial, it’s now a part
+of the base TOM Toolkit, but all of the information here should provide
+you with the ability to write your own.
diff --git a/docs/customization/customize_templates.md b/docs/customization/customize_templates.md
deleted file mode 100644
index b01c36b6f..000000000
--- a/docs/customization/customize_templates.md
+++ /dev/null
@@ -1,152 +0,0 @@
-Customizing TOM Templates
--------------------------
-
-So you've got a TOM up and running, and your homepage looks something like this:
-
-
-
-This is fine for starting out, but since you're running a TOM for a specific
-project, the homepage ought to reflect that.
-
-If you haven't already, please read through the [Getting Started](/introduction/getting_started)
-docs and return here when you have a project layout that looks something like this:
-
-
-```
-mytom
-├── db.sqlite3
-├── manage.py
-└── mytom
- ├── __init__.py
- ├── settings.py
- ├── urls.py
- └── wsgi.py
-```
-
-We are going to override the html template included with the TOM Toolkit, `tom_common/index.html`,
-so that we can edit some text and change the image. Overriding and extending templates is
-[documented extensively](https://docs.djangoproject.com/en/2.1/howto/overriding-templates/) on
-Django's website and we highly recommend reading these docs if you plan on customizing your
-TOM further.
-
-Since the template we want to override is already part of the TOM Toolkit source
-code, we can use it as a starting point for our customized template. In fact,
-we'll copy and paste the entire thing from the [source code of TOM Toolkit](https://github.com/TOMToolkit/tom_base/blob/master/tom_common/templates/tom_common/index.html).
-and place it in our project. The template we are looking for is `tom_common/index.html`
-
-Let's download and copy that template into our `templates` folder
-(including the `tom_common` sub-directory) so that our directory structure now
-looks like this:
-
-```
-├── db.sqlite3
-├── manage.py
-├── templates
-│ └── tom_common
-│ └── index.html
-└── mytom
- ├── __init__.py
- ├── settings.py
- ├── urls.py
- └── wsgi.py
-```
-
-Now let's make a few changes to the `templates/tom_common/index.html` template:
-
-```html
-{% extends 'tom_common/base.html' %}
-{% load static targets_extras observation_extras dataproduct_extras tom_common_extras %}
-{% block title %}Home{% endblock %}
-{% block content %}
-
-
- Project LEO
-
-
-
- 
- Project LEO is a very serious survey of the most important constellation.
-
-
-
- Next steps
-
- Other Resources
-
-
-
-
-
- {% recent_comments %}
-
-
-
-
- {% recent_targets %}
-
-
-{% endblock %}
-```
-Look for the block of HTML we changed between the <\!-- BEGIN MODIFIED CONTENT -->
-and <\!-- END MODIFIED CONTENT --> comments. Everything else is the same as the
-base template.
-
-We've just changed a few lines of HTML, but basically left the template alone. Reload your homepage,
-and you should see something like this:
-
-
-
-Thats it! You've just customized your TOM homepage.
-
-### Using static files
-
-Instead of linking to an image hosted online already, we can display static files
-in our project directly. For this we will use [Django's static
-files](https://docs.djangoproject.com/en/2.1/howto/static-files/) capabilities.
-
-If you ran the tom_setup script, you should have a directory `static` at the top
-level of your project. Within this folder, make a directory `img`. In this folder,
-place an image you'd like to display on your homepage. For example, `mytom.jpg`.
-
- cp mytom.jpg static/img/
-
-Now let's edit our template to use Django's `static` template tag to display the
-image:
-
-```html
-{% raw %}
- 
-{% endraw %}
-```
-
-After reloading the page, you should now see `mytom.jpg` displayed instead of the
-remote cat image.
-
-### Further Reading
-
-Any template included in the TOM Toolkit (or any other Django app) can be customized. Please
-see the [official Django docs](https://docs.djangoproject.com/en/2.1/howto/overriding-templates/)
-for more details.
diff --git a/docs/customization/customize_templates.rst b/docs/customization/customize_templates.rst
new file mode 100644
index 000000000..49b35346c
--- /dev/null
+++ b/docs/customization/customize_templates.rst
@@ -0,0 +1,170 @@
+Customizing TOM Templates
+-------------------------
+
+So you’ve got a TOM up and running, and your homepage looks something
+like this:
+
+|image0|
+
+This is fine for starting out, but since you’re running a TOM for a
+specific project, the homepage ought to reflect that.
+
+If you haven’t already, please read through the `Getting
+Started `__ docs and return here when you
+have a project layout that looks something like this:
+
+::
+
+ mytom
+ ├── db.sqlite3
+ ├── manage.py
+ └── mytom
+ ├── __init__.py
+ ├── settings.py
+ ├── urls.py
+ └── wsgi.py
+
+We are going to override the html template included with the TOM
+Toolkit, ``tom_common/index.html``, so that we can edit some text and
+change the image. Overriding and extending templates is `documented
+extensively `__
+on Django’s website and we highly recommend reading these docs if you
+plan on customizing your TOM further.
+
+Since the template we want to override is already part of the TOM
+Toolkit source code, we can use it as a starting point for our
+customized template. In fact, we’ll copy and paste the entire thing from
+the `source code of TOM
+Toolkit `__.
+and place it in our project. The template we are looking for is
+``tom_common/index.html``
+
+Let’s download and copy that template into our ``templates`` folder
+(including the ``tom_common`` sub-directory) so that our directory
+structure now looks like this:
+
+::
+
+ ├── db.sqlite3
+ ├── manage.py
+ ├── templates
+ │ └── tom_common
+ │ └── index.html
+ └── mytom
+ ├── __init__.py
+ ├── settings.py
+ ├── urls.py
+ └── wsgi.py
+
+Now let’s make a few changes to the ``templates/tom_common/index.html``
+template:
+
+.. code:: html
+
+ {% extends 'tom_common/base.html' %}
+ {% load static targets_extras observation_extras dataproduct_extras tom_common_extras %}
+ {% block title %}Home{% endblock %}
+ {% block content %}
+
+
+ Project LEO
+
+
+
+
+ Project LEO is a very serious survey of the most important constellation.
+
+
+
+ Next steps
+
+ Other Resources
+
+
+
+
+
+ {% recent_comments %}
+
+
+
+
+ {% recent_targets %}
+
+
+ {% endblock %}
+
+Look for the block of HTML we changed between the and comments. Everything else is
+the same as the base template.
+
+We’ve just changed a few lines of HTML, but basically left the template
+alone. Reload your homepage, and you should see something like this:
+
+|image1|
+
+Thats it! You’ve just customized your TOM homepage.
+
+Using static files
+~~~~~~~~~~~~~~~~~~
+
+Instead of linking to an image hosted online already, we can display
+static files in our project directly. For this we will use `Django’s
+static
+files `__
+capabilities.
+
+If you ran the tom_setup script, you should have a directory ``static``
+at the top level of your project. Within this folder, make a directory
+``img``. In this folder, place an image you’d like to display on your
+homepage. For example, ``mytom.jpg``.
+
+::
+
+ cp mytom.jpg static/img/
+
+Now let’s edit our template to use Django’s ``static`` template tag to
+display the image:
+
+.. code:: html
+
+ {% raw %}
+
+ {% endraw %}
+
+After reloading the page, you should now see ``mytom.jpg`` displayed
+instead of the remote cat image.
+
+Further Reading
+~~~~~~~~~~~~~~~
+
+Any template included in the TOM Toolkit (or any other Django app) can
+be customized. Please see the `official Django
+docs `__
+for more details.
+
+.. |image0| image:: /_static/customize_templates_doc/tomhomepagenew.png
+.. |image1| image:: /_static/customize_templates_doc/tomhomepagemod.png
diff --git a/docs/customization/customizing_data_processing.md b/docs/customization/customizing_data_processing.md
deleted file mode 100644
index fea329b95..000000000
--- a/docs/customization/customizing_data_processing.md
+++ /dev/null
@@ -1,157 +0,0 @@
-Customizing Data Processing
----------------------------
-
-One of the many goals of the TOM Toolkit is to enable the simplification of the flow of your data from observations. To
-that end, there's some built-in functionality that can be overridden to allow your TOM to work for your use case.
-
-To begin, here's a brief look at part of the structure of the tom_dataproducts app in the TOM Toolkit:
-
-```
-tom_dataproducts
-├──hooks.py
-├──models.py
-└──processors
- ├──data_serializers.py
- ├──photometry_processor.py
- └──spectroscopy_processor.py
-```
-
-Let's start with a quick overview of `models.py`. The file contains the Django models for the dataproducts app--in our
-case, `DataProduct` and `ReducedDatum`. The `DataProduct` contains information about uploaded or saved `DataProducts`,
-such as the file name, file path, and what kind of file it is. The `ReducedDatum` contains individual science data
-points that are taken from the `DataProduct` files. Examples of `ReducedDatum` points would be individual photometry
-points or individual spectra.
-
-Each `DataProduct` also has a `data_product_type`. The `data_product_type` is simply a description of what the file is,
-more or less, and is customizable. The list of supported `data_product_type`s is maintained in `settings.py`:
-
-```python
-# Define the valid data product types for your TOM. Be careful when removing items, as previously valid types will no
-# longer be valid, and may cause issues unless the offending records are modified.
-DATA_PRODUCT_TYPES = {
- 'photometry': ('photometry', 'Photometry'),
- 'fits_file': ('fits_file', 'FITS File'),
- 'spectroscopy': ('spectroscopy', 'Spectroscopy'),
- 'image_file': ('image_file', 'Image File')
-}
-```
-
-In order to add new data product types, simply add a new key/value pair, with the value being a 2-tuple. The first
-tuple item is the database value, and the second is the display value.
-
-All data products are automatically "processed" on upload, as well. Of course, that can mean different things to
-different TOMs! The TOM has two built-in data processors, both of which simply ingest the data into the database,
-and those are also specified in `settings.py`:
-
-```python
-DATA_PROCESSORS = {
- 'photometry': 'tom_dataproducts.processors.photometry_processor.PhotometryProcessor',
- 'spectroscopy': 'tom_dataproducts.processors.spectroscopy_processor.SpectroscopyProcessor',
-}
-```
-
-When a user either uploads a `DataProduct` to their TOM, the TOM runs `process_data()` from the corresponding
-`DataProcessor` subclass specified in `DATA_PROCESSORS` seen above. To illustrate, this is the base `DataProcessor`
-class:
-
-```python
-import mimetypes
-
-...
-
-class DataProcessor():
-
- FITS_MIMETYPES = ['image/fits', 'application/fits']
- PLAINTEXT_MIMETYPES = ['text/plain', 'text/csv']
-
- mimetypes.add_type('image/fits', '.fits')
- mimetypes.add_type('image/fits', '.fz')
- mimetypes.add_type('application/fits', '.fits')
- mimetypes.add_type('application/fits', '.fz')
-
- def process_data(self, data_product):
- pass
-
-```
-
-Now let's look at the built-in data processors. First, let's check out the `PhotometryProcessor`, which inherits from
-`DataProcessor`:
-
-```python
-class PhotometryProcessor(DataProcessor):
-
- def process_data(self, data_product):
- mimetype = mimetypes.guess_type(data_product.data.path)[0]
- if mimetype in self.PLAINTEXT_MIMETYPES:
- photometry = self._process_photometry_from_plaintext(data_product)
- return [(datum.pop('timestamp'), json.dumps(datum)) for datum in photometry]
- else:
- raise InvalidFileFormatException('Unsupported file type')
-```
-
-This class has an implementation of `process_data()` from the superclass `DataProcessor`. The implementation calls an
-internal method `_process_photometry_from_plaintext()`, which return a `list` of `dict`s. Each dict contains the values
-for the timestamp, magnitude, filter, and error for that photometry point. The list is then transformed into a list of
-2-tuples, with the first value being the photometry timestamp, and the second being the JSON-ified remaining values.
-
-Next, let's look at the `SpectroscopyProcessor`:
-
-```python
-class SpectroscopyProcessor(DataProcessor):
-
- DEFAULT_WAVELENGTH_UNITS = units.angstrom
- DEFAULT_FLUX_CONSTANT = units.erg / units.cm ** 2 / units.second / units.angstrom
-
- def process_data(self, data_product):
-
- mimetype = mimetypes.guess_type(data_product.data.path)[0]
- if mimetype in self.FITS_MIMETYPES:
- spectrum, obs_date = self._process_spectrum_from_fits(data_product)
- elif mimetype in self.PLAINTEXT_MIMETYPES:
- spectrum, obs_date = self._process_spectrum_from_plaintext(data_product)
- else:
- raise InvalidFileFormatException('Unsupported file type')
-
- serialized_spectrum = SpectrumSerializer().serialize(spectrum)
-
- return [(obs_date, serialized_spectrum)]
-```
-
-Just like the `PhotometryProcessor`, this class inherits from `DataProcessor` and implements `process_data()`. This is a
-requirement for a custom DataProcessor! This `process_data()` method handles two file types, unlike the previous
-example, each of which calls an internal method that returns a `Spectrum1D` object. Again, like the
-`PhotometryProcessor`, a list of 2-tuples is created, with the first value being the timestamp, and the second being
-the JSON spectrum.
-
-You may be wondering why these two methods return lists of 2-tuples, especially when the `SpectroscopyProcessor` only
-returns a list of length one. The rationale is to ensure that you, the TOM user, shouldn't have to worry about the
-database insertion, so the internal logic handles that aspect, and it can do so whether you return one data point or
-many data points.
-
-For a custom `DataProcessor`, there are just a few required steps. The first is to create a class that implements
-`DataProcessor`, like so:
-
-```python
-from tom_dataproducts.data_processor import DataProcessor
-
-
-class MyDataProcessor(DataProcessor):
-
- def process_data(self, data_product):
- # custom data processing here
-
- return [(timestamp1, json1), (timestamp2, json2), ..., (timestampN, dictN)]
-```
-
-Let's say that this file lives at `mytom/my_data_processor.py`. Now the processor needs to be added to
-`DATA_PROCESSORS`, and it can either process a new data product type, or replace an existing one. Let's replace
-spectroscopy:
-
-```python
-DATA_PROCESSORS = {
- 'photometry': 'tom_dataproducts.processors.photometry_processor.PhotometryProcessor',
- 'spectroscopy': 'mytom.my_data_processor.MyDataProcessor',
-}
-```
-
-And that's it! Now your TOM will run the data processing specific to your case instead of the default one.
diff --git a/docs/customization/customsettings.md b/docs/customization/customsettings.md
deleted file mode 100644
index 9646fe6bc..000000000
--- a/docs/customization/customsettings.md
+++ /dev/null
@@ -1,149 +0,0 @@
-TOM Specific Settings
----------------------
-
-The following is a list of TOM Specific settings to be added/edited in your
-project's `settings.py`. For explanations of Django specific settings, see the
-[official documentation](https://docs.djangoproject.com/en/2.1/ref/settings/).
-
-### [AUTH_STRATEGY](#auth_strategy)
-
-Default: 'READ_ONLY'
-
-Determines how your TOM treats unauthenticated users. A value of **READ_ONLY**
-allows unauthenticated users to view most pages on your TOM, but not to change
-anything. A value of **LOCKED** requires all users to login before viewing any
-page. Use the [**OPEN_URLS**](#open_urls) setting for adding exemptions.
-
-
-### [DATA_PRODUCT_TYPES](#data_types)
-
-Default:
-
- {
- 'spectroscopy': ('spectroscopy', 'Spectroscopy'),
- 'photometry': ('photometry', 'Photometry')
- }
-
-A list of machine readable, human readable tuples which determine the choices
-available to categorize reduced data.
-
-
-### [EXTRA_FIELDS](#extra_fields)
-
-Default: []
-
-A list of extra fields to add to your targets. These can be used if the predefined
-target fields do not match your needs. Please see the documentation on [Adding
-Custom Fields to Targets](/customization/target_fields) for an explanation of how to use
-this feature.
-
-
-### [FACILITIES](#facilities)
-
-Default:
-
- {
- 'LCO': {
- 'portal_url': 'https://observe.lco.global',
- 'api_key': os.getenv('LCO_API_KEY', ''),
- }
- }
-
-Observation facilities read their configuration values from this dictionary.
-Although each facility is different, if you plan on using one you'll probably have
-to configure it here first. For example the LCO facility requires you to provide a
-value for the `api_key` configuration value.
-
-
-### [HINTS](#hints)
-
-Default:
-
-HINTS_ENABLED = False
-HINT_LEVEL = 20
-
-A few messages are sprinkled throughout the TOM Toolkit that offer suggestions on
-things you might want to change right out of the gate. These can be turned on and
-off, and the level adjusted. For more information on Django message levels, see
-the [Django messages framework documentation](https://docs.djangoproject.com/en/2.2/ref/contrib/messages/#message-levels).
-
-
-### [HOOKS](#hooks)
-
-Default:
-
- {
- 'target_post_save': 'tom_common.hooks.target_post_save',
- 'observation_change_state': 'tom_common.hooks.observation_change_state',
- 'data_product_post_upload': 'tom_dataproducts.hooks.data_product_post_upload',
- }
-
-A dictionary of action, method code hooks to run. These hooks allow running
-arbitrary python code when specific actions happen within a TOM, such as an
-observation changing state. See the documentation on [Running Custom Code on
-Actions in your TOM](/advanced/custom_code) for more details and available hooks.
-
-
-### [OPEN_URLS](#open_urls)
-
-Default: []
-
-With an [**AUTH_STRATEGY**](#auth_strategy) value of **LOCKED**, urls in this list will remain
-visible to unauthenticated users. You might add the homepage ('/'), for example.
-
-
-### [TARGET_TYPE](#target_type)
-
-Default: No default
-
-Can be either **SIDEREAL** or **NON_SIDEREAL**. This settings determines the
-default target type for your TOM. TOMs can still create and work with targets of
-both types even after this option is set, but setting it to one of the values will
-optimize the workflow for that target type.
-
-
-### [TOM_ALERT_CLASSES](#tom_alert_classes)
-
-Default:
-
- [
- 'tom_alerts.brokers.mars.MARSBroker',
- 'tom_alerts.brokers.lasair.LasairBroker',
- 'tom_alerts.brokers.scout.ScoutBroker'
- ]
-
-A list of tom alert classes to make available to your TOM. If you have written or
-downloaded additional alert classes you would make them available here. If you'd
-like to write your own alert module please see the documentation on [Creating an
-Alert Module for the TOM Toolkit](/customization/create_broker).
-
-
-### [TOM_FACILITY_CLASSES](#tom_facility_classes)
-
-Default:
-
- [
- 'tom_observations.facilities.lco.LCOFacility',
- 'tom_observations.facilities.gemini.GEMFacility',
- ]
-
-A list of observation facility classes to make available to your TOM. If you have
-written or downloaded a custom observation facility you would add the class to
-this list to make your TOM load it.
-
-
-### [TOM_HARVESTER_CLASSES](#tom_harvester_classes)
-
-Default:
-
- [
- 'tom_catalogs.harvesters.simbad.SimbadHarvester',
- 'tom_catalogs.harvesters.ned.NEDHarvester',
- 'tom_catalogs.harvesters.jplhorizons.JPLHorizonsHarvester',
- 'tom_catalogs.harvesters.mpc.MPCHarvester',
- 'tom_catalogs.harvesters.tns.TNSHarvester',
- ]
-
-A list of TOM harverster classes to make available to your TOM. If you have
-written or downloaded additional harvester classes you would make them available
-here.
diff --git a/docs/customization/index.rst b/docs/customization/index.rst
index b64aa7c83..87882b1f7 100644
--- a/docs/customization/index.rst
+++ b/docs/customization/index.rst
@@ -1,26 +1,16 @@
-***********
-Customizing
-***********
+Customization
+=============
.. toctree::
- :maxdepth: 1
+ :maxdepth: 2
:hidden:
- customsettings
customize_templates
adding_pages
- target_fields
- customizing_data_processing
- create_broker
- customize_observations
- plotting_data
- permissions
- automation
+ customize_template_tags
-Start here to learn how to customize the look and feel of your TOM or add new functionality.
-:doc:`Custom Settings ` - Settings available to the TOM Toolkit which you may want to
-configure.
+Start here to learn how to customize the look and feel of your TOM or add new functionality.
:doc:`Customizing TOM Templates ` - Learn how to override built in TOM templates to
change the look and feel of your TOM.
@@ -28,23 +18,5 @@ change the look and feel of your TOM.
:doc:`Adding new Pages to your TOM ` - Learn how to add entirely new pages to your TOM,
displaying static html pages or dynamic database-driven content.
-:doc:`Adding Custom Target Fields ` - Learn how to add custom fields to your TOM Targets if the
-defaults do not suffice.
-
-:doc:`Adding Custom Data Processing ` - Learn how you can process data into your
-TOM from uploaded data products.
-
-:doc:`Building a TOM Alert Broker ` - Learn how to build an Alert Broker module to add new
-sources of targets to your TOM.
-
-:doc:`Changing Request Submission Behavior ` - Learn how to customize the LCO
-Observation Module in order to add additional parameters to observation requests sent to the LCO Network.
-
-:doc:`Creating Plots from TOM Data ` - Learn how to create plots using plot.ly and your TOM
-data to display anywhere in your TOM.
-
-:doc:`The Permissions System ` - Use the permissions system to limit access to targets in your
-TOM.
-
-:doc:`Automating Tasks ` - Run commands automatically to keep your TOM working even when you
-aren’t
\ No newline at end of file
+:doc:`Customizing Template Tags ` - Learn how to write your own template tags to display
+the data you need.
diff --git a/docs/customization/plotting_data.md b/docs/customization/plotting_data.md
deleted file mode 100644
index 83f9cb235..000000000
--- a/docs/customization/plotting_data.md
+++ /dev/null
@@ -1,179 +0,0 @@
-Plotting Data
--------------
-
-The TOM Toolkit provides a few basic plots, such as photometry, spectroscopy and
-target distribution. Sometimes it would be useful to visualize data in a different
-way.
-
-In this tutorial you will learn how to build and display a very simple plot
-in our TOM: number of reduced data per target. The end result will demonstrate
-how to create a [plot.ly](https://plot.ly) plot with data from our TOM. You will
-even package the code in it's own app so we can share it with other TOM users that
-might find it useful.
-
-If you haven't already read the documentation on [customizing
-templates](/customization/customize_templates) you should read it first. You'll need to
-edit a template in order to view your new plot somewhere.
-
-First, start a new app in our project to house the new plot (and perhaps
-other additions!):
-
- ./manage.py startapp myplots
-
-This will create a new [Django
-app](https://docs.djangoproject.com/en/2.1/intro/tutorial01/#creating-the-polls-app) in your project
-named myplots:
-
- myplots
- ├── admin.py
- ├── apps.py
- ├── __init__.py
- ├── migrations
- │ └── __init__.py
- ├── models.py
- ├── tests.py
- └── views.py
-
- 1 directory, 7 files
-
-Note you don't necessarily have to start a new app. If you've already started an
-app that you'd like to reuse, that works too.
-
-Now install the new app into your project's settings.py file:
-
-```python
-INSTALLED_APPS = [
- 'django.contrib.admin',
- 'django.contrib.auth',
- ...
- 'myplots',
-]
-```
-
-Now that the `myplots` app is installed, create the directories necessary to
-contain your new plot:
-
- mkdir -p myplots/templates/myplots
- mkdir myplots/templatetags
-
-The templates directory will contain the html template you can include in other
-templates to display your plot. The templatetags directory will contain the python
-code to construct the plot.ly plot.
-
-Start by creating the
-[templatetags](https://docs.djangoproject.com/en/2.1/howto/custom-template-tags/)
-file:
-
- touch myplots/templatetags/myplots_tags.py
-
-Edit this file, starting with the necessary imports:
-
-```python
-from plotly import offline
-import plotly.graph_objs as go
-from django import template
-
-from tom_targets.models import Target
-```
-
-The `plotly` imports are needed for building an offline plot. The django
-`template` import gives access to the template library, which will allow for
-registering the template tag. Finally, the TOM Toolkit `Target` class will allow
-access to the `Target` model (for querying).
-
-Next, add the boiler plate code for a template tag:
-
-```python
-register = template.Library()
-
-
-@register.inclusion_tag('myplots/targets_reduceddata.html')
-def targets_reduceddata(targets=Target.objects.all()):
-```
-
-First we instantiate the `register` decorator. You don't need to know much about
-this other that it allows us to register functions as templatetags. The function
-`targets_reduceddata` is decorated with the `register` decorator, which takes as
-an argument the template to render. The function definition takes in a queryset of
-`Target`s as a keyword argument, but if none are supplied, defaults to all `Target`s
-in the database.
-
-Next, add the function body:
-
-```python
- # order targets by creation date
- targets = targets.order_by('-created')
- # x axis: target names. y axis: datum count
- data = [go.Bar(
- x=[target.name for target in targets],
- y=[target.reduceddatum_set.count() for target in targets]
- )]
- # Create the plot
- figure = offline.plot(go.Figure(data=data), output_type='div', show_link=False)
- # Add plot to the template context
- return {'figure': figure}
-```
-
-As the comments describe, the function code iterates over each `Target` in the
-`targets` queryset adding the target name and datum count as x/y values to the
-`Bar` data structure. Check out the [plot.ly bar chart
-documentation](https://plot.ly/python/bar-charts/) for more information about the
-options available to you. As an exercise, try changing the values in the y axis.
-Or you could use a different chart type.
-
-Finally, the code adds the plot.ly plot to the template rendering context. Next we
-will create this template where this context will be rendered.
-
-Create the file, making sure it matches the template name specified in the
-template tag definition beforehand:
-
- touch myplots/templates/myplots/targets_reduceddata.html
-
-This file contains the simple contents:
-
- {% raw %}
- {{ figure|safe }}
- {% endraw %}
-
-All this template does is output the `figure` variable, which is the html
-generated from plotly in the templatetag. We also tell django that the output is
-safe, so that it doesn't escape the html. That's it.
-
-**Note:** If you're running the development server, restart it now. Django doesn't
-automatically pick up new templatetags.
-
-Now that the templatetag and template are complete, we can use it in any template.
-You might have your own templates which you'd like to add the plot to, or perhaps
-you've customized one of the TOM supplied templates as per the [customizing
-templates](/customization/customize_templates) documentation. Either way, including the
-templatetag works the same way. At the top of the template (after any 'extends')
-load the new tag library:
-
- {% raw %}
- {% load myplots_tags %}
- {% endraw %}
-
-Now insert the templatetag somewhere in the template where you'd like it to
-appear:
-
- {% raw %}
- {% targets_reduceddata %}
- {% endraw %}
-
-If your parent template already has a queryset of targets available in the context
-(for example, a target list page) you can pass it in to be used in your plot:
-
- {% raw %}
- {% targets_reduceddata targets %}
- {% endraw %}
-
-Otherwise the plot will simply use all targets in your database. Either way, you
-should end up with something like this:
-
-
-
-That's it! Plot.ly provides a wide range of plotting capabilities, you should
-reference [the documentation](https://plot.ly/python/) for more information. It
-would also be helpful to read [Django's
-ORM](https://docs.djangoproject.com/en/2.1/topics/db/) to become familiarized with
-wide range of methods of querying data.
diff --git a/docs/customization/target_fields.md b/docs/customization/target_fields.md
deleted file mode 100644
index dbf3838e7..000000000
--- a/docs/customization/target_fields.md
+++ /dev/null
@@ -1,134 +0,0 @@
-Adding Custom Fields to Targets
----
-
-
-Sometimes you'd like to store data for targets but the predefined fields that the
-TOM Toolkit provides aren't enough. The TOM Toolkit allows you to define extra
-fields for your targets so you can associate different kinds of data with them.
-For example, you might be studying high redshift galaxies. In this case, it would
-make sense to be able to store the redshift of your targets. You could then do a
-search for targets with a redshift less than or greater than a particular value,
-or use the redshift value to make decisions in your science code.
-
-**Note**: There is a performance hit when using extra fields. Try to use the
-built in fields whenever possible.
-
-### Enabling extra fields
-
-To start, find the `EXTRA_FIELDS` definition in your `settings.py`:
-
-```python
-# Define extra target fields here. Types can be any of "number", "string", "boolean" or "datetime"
-# For example:
-# EXTRA_FIELDS = [
-# {'name': 'redshift', 'type': 'number'},
-# {'name': 'discoverer', 'type': 'string'}
-# {'name': 'eligible', 'type': 'boolean'},
-# {'name': 'dicovery_date', 'type': 'datetime'}
-# ]
-EXTRA_FIELDS = []
-```
-
-We can define any number of extra fields in the array. Each item in the array
-is a dictionary with two values: name and type. Name is simply what you would like
-to name your field. Type is the datatype of the field and can be one of: `number`,
-`string`, `boolean` or `datetime`. These types allow the TOM Toolkit to properly
-store, filter and display these values elsewhere.
-
-As an example, let's change the setting to look like this:
-
-```python
- EXTRA_FIELDS = [
- {'name': 'redshift', 'type': 'number'},
- ]
-```
-
-This will make an extra field with the name "redshift" and a type of "number"
-available to add to our targets.
-
-### Using extra fields
-
-Now if you go to the target creation page, you should see the new field available:
-
-
-
-And if we go to our list of targets, we should see redshift as a field available
-to filter on:
-
-
-
-Extra fields with the `number` type allow filtering on range of values. The same
-goes for fields with the `datetime` type. `string` types to a case insensitive
-inclusive search, and `boolean` fields to a simple matching comparison.
-
-Of course, redshift does appear on our target's display page as well:
-
-
-
-To hide extra fields from the target page, we can set the "hidden" key (this
-doesn't affect filtering and searching):
-
-```python
- EXTRA_FIELDS = [
- {'name': 'redshift', 'type': 'number', 'hidden': True},
- ]
-```
-
-And we can set a default value for an extra field by including a default key/value pair:
-
-```python
- EXTRA_FIELDS = [
- {'name': 'redshift', 'type': 'number', 'default': 0},
- ]
-```
-
-### Displaying extra fields in templates
-
-If we want to display the redshift in other places, we can use a template filter to
-do that. For example, we might want to display the redshift value in the target
-list table.
-
-At the top of our template make sure to load `targets_extras`:
-
-```
-{% raw %}
- {% load targets_extras %}
-{% endraw %}
-```
-
-Now we can use the `target_extra_field` filter wherever a target object is
-available in the template context:
-
-```
-{% raw %}
- {{ target|target_extra_field:"redshift" }}
-{% endraw %}
-```
-
-The result is the redshift value being printed on the template:
-
-
-
-### Working with extra fields programatically
-
-If you'd like to update or save extra fields to your targets in code, there are a
-few methods you can use. The simplest is to simply pass in a dictionary of extra data to your
-target's `save()` method using the `extras` keyword argument:
-
-```python
-target = Target.objects.get(name='example')
-target.save(extras={'foo': 42})
-```
-
-The example target above will now have an extra field "foo" with the value 42.
-
-For more precise control, you can access `TargetExtra` models directly. To remove
-an extra, for example:
-
-```python
-target = Target.objects.get(name='example')
-target_extra = target.targetextra_set.get(key='foo')
-target_extra.delete()
-```
-
-The above deleted the target extra on a target with the key of "foo".
diff --git a/docs/deployment/amazons3.md b/docs/deployment/amazons3.md
deleted file mode 100644
index 8203c965f..000000000
--- a/docs/deployment/amazons3.md
+++ /dev/null
@@ -1,108 +0,0 @@
-Using Amazon S3 to Store Data for a TOM
----
-
-If a TOM needs to store a large amount of data, like images or spectra, it may
-eventually become impractical to do so on a local hard drive or network share.
-This is where cloud storage services like [Amazon S3](https://aws.amazon.com/s3/)
-come in handy. These services allow you to store large quantities of data at a low
-cost, while providing high reliability and feature rich services. In most cases
-using a cloud storage system also provides performance and speed increases to your
-application.
-
-Configuring the TOM toolkit to store data on Amazon S3 is fairly straightforward.
-Once enabled, data product downloads, uploads, and static assets (images,
-stylesheets, etc) will be stored in Amazon S3 instead of the local filesystem
-where your TOM is run.
-
-### Sign up for an AWS Account
-
-To use S3, you'll first need to sign up for an [Amazon Web
-Services](https://portal.aws.amazon.com/billing/signup#/start) account. New
-accounts get access to one year of free tier access which includes a year of S3 at
-a max of 5GB. If you're interested in the cost beyond 5Gb, try out the [Amazon
-cost calculator](https://calculator.s3.amazonaws.com/index.html).
-
-Once you have created an account, you'll need your access key id and secret access
-key. These can be found under your profile settings -> "My security credentials".
-Make sure you save these in a safe place, you'll need them later.
-
-### Create a bucket
-
-A bucket is like the highest level folder you can store data in S3. You should
-create one for your TOM. Name it whatever you'd like. Most of the default settings
-should be fine.
-
-**We need to enable CORS** for JS9 (or any other javascript code that wants to
-access our data directly) to work. Under the "Permissions" tab for your bucket,
-find the section for "CORS configuration". In the editor, paste the following policy:
-
-```xml
-
-
-
- *
- GET
- *
-
-
-```
-
-This policy allows GET requests from anywhere. Feel free to edit it to match your
-particular use case specifically.
-
-
-### Configure S3 Storage backend for your TOM
-
-Now that we have a bucket set up, let's configure our TOM to use it. First we need
-to install two additional python packages. You should add these to your project's
-`requirements.txt`.:
-
-* django-storages
-* boto3
-
-Next, we'll edit our TOM's `settings.py` to use S3 instead of local storage. Place
-the following lines somewhere around the existing static files configuration
-settings:
-
-```python
-DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
-STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
-
-AWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID', '')
-AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRECT_ACCESS_KEY', '')
-AWS_STORAGE_BUCKET_NAME = os.getenv('AWS_STORAGE_BUCKET_NAME', '')
-AWS_S3_REGION_NAME = os.getenv('AWS_S3_REGION_NAME', '')
-AWS_DEFAULT_ACL = None
-```
-
-Notice that these settings get their values via environmental variables. Depending
-on how you deploy your TOM, you can set these in a variety of ways. For example:
-export AWS_ACCESS_KEY_ID=MyAccessKey would be one way to set them using Bash.
-
-* AWS_ACCESS_KEY_ID is your access key id from your security credentials.
-* AWS_SECRECT_ACCESS_KEY is your secret access key from your security credentials.
-* AWS_STORAGE_BUCKET_NAME is the name you gave to the bucket you created.
-* AWS_S3_REGION_NAME is the name of th region you created your bucket in.
-
-Once these settings are filled out, your TOM should store all future data in S3.
-If you had existing data in your TOM, you should copy it over to your bucket in
-the exact same way it was stored locally.
-
-### For Heroku Users
-
-If you are using Heroku (perhaps by following the [Heroku deployment
-guide](https://tomtoolkit.github.io/docs/deployment_heroku)) there is one more
-additional step. At the very bottom of `settings.py` change the line:
-
- django_heroku.settings(locals())
-
-to:
-
- django_heroku.settings(locals(), staticfiles=False)
-
-This instructs the `django-heroku` package to not automatically configure static
-files for your TOM (since we are explicitly using S3 now).
-
-Additionally, Heroku makes it easy to set environmental variables.
-See [Configuration and Config Vars](
-https://devcenter.heroku.com/articles/config-vars).
diff --git a/docs/deployment/amazons3.rst b/docs/deployment/amazons3.rst
new file mode 100644
index 000000000..2b90012a8
--- /dev/null
+++ b/docs/deployment/amazons3.rst
@@ -0,0 +1,126 @@
+Using Amazon S3 to Store Data for a TOM
+---------------------------------------
+
+If a TOM needs to store a large amount of data, like images or spectra,
+it may eventually become impractical to do so on a local hard drive or
+network share. This is where cloud storage services like `Amazon
+S3 `__ come in handy. These services allow
+you to store large quantities of data at a low cost, while providing
+high reliability and feature rich services. In most cases using a cloud
+storage system also provides performance and speed increases to your
+application.
+
+Configuring the TOM toolkit to store data on Amazon S3 is fairly
+straightforward. Once enabled, data product downloads, uploads, and
+static assets (images, stylesheets, etc) will be stored in Amazon S3
+instead of the local filesystem where your TOM is run.
+
+Sign up for an AWS Account
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+To use S3, you’ll first need to sign up for an `Amazon Web
+Services `__
+account. New accounts get access to one year of free tier access which
+includes a year of S3 at a max of 5GB. If you’re interested in the cost
+beyond 5Gb, try out the `Amazon cost
+calculator `__.
+
+Once you have created an account, you’ll need your access key id and
+secret access key. These can be found under your profile settings -> “My
+security credentials”. Make sure you save these in a safe place, you’ll
+need them later.
+
+Create a bucket
+~~~~~~~~~~~~~~~
+
+A bucket is like the highest level folder you can store data in S3. You
+should create one for your TOM. Name it whatever you’d like. Most of the
+default settings should be fine.
+
+**We need to enable CORS** for JS9 (or any other javascript code that
+wants to access our data directly) to work. Under the “Permissions” tab
+for your bucket, find the section for “CORS configuration”. In the
+editor, paste the following policy:
+
+.. code:: xml
+
+
+
+
+ *
+ GET
+ *
+
+
+
+This policy allows GET requests from anywhere. Feel free to edit it to
+match your particular use case specifically.
+
+Configure S3 Storage backend for your TOM
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now that we have a bucket set up, let’s configure our TOM to use it.
+First we need to install two additional python packages. You should add
+these to your project’s ``requirements.txt``.:
+
+- django-storages
+- boto3
+
+Next, we’ll edit our TOM’s ``settings.py`` to use S3 instead of local
+storage. Place the following lines somewhere around the existing static
+files configuration settings:
+
+.. code:: python
+
+ DEFAULT_FILE_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
+ STATICFILES_STORAGE = 'storages.backends.s3boto3.S3Boto3Storage'
+
+ AWS_ACCESS_KEY_ID = os.getenv('AWS_ACCESS_KEY_ID', '')
+ AWS_SECRET_ACCESS_KEY = os.getenv('AWS_SECRECT_ACCESS_KEY', '')
+ AWS_STORAGE_BUCKET_NAME = os.getenv('AWS_STORAGE_BUCKET_NAME', '')
+ AWS_S3_REGION_NAME = os.getenv('AWS_S3_REGION_NAME', '')
+ AWS_DEFAULT_ACL = None
+
+Notice that these settings get their values via environmental variables.
+Depending on how you deploy your TOM, you can set these in a variety of
+ways. For example: export AWS_ACCESS_KEY_ID=MyAccessKey would be one way
+to set them using Bash.
+
+- AWS_ACCESS_KEY_ID is your access key id from your security
+ credentials.
+- AWS_SECRECT_ACCESS_KEY is your secret access key from your security
+ credentials.
+- AWS_STORAGE_BUCKET_NAME is the name you gave to the bucket you
+ created.
+- AWS_S3_REGION_NAME is the name of th region you created your bucket
+ in.
+
+Once these settings are filled out, your TOM should store all future
+data in S3. If you had existing data in your TOM, you should copy it
+over to your bucket in the exact same way it was stored locally.
+
+For Heroku Users
+~~~~~~~~~~~~~~~~
+
+If you are using Heroku (perhaps by following the `Heroku deployment
+guide `__) there is
+one more additional step. At the very bottom of ``settings.py`` change
+the line:
+
+::
+
+ django_heroku.settings(locals())
+
+to:
+
+::
+
+ django_heroku.settings(locals(), staticfiles=False)
+
+This instructs the ``django-heroku`` package to not automatically
+configure static files for your TOM (since we are explicitly using S3
+now).
+
+Additionally, Heroku makes it easy to set environmental variables. See
+`Configuration and Config
+Vars `__.
diff --git a/docs/deployment/deployment_heroku.md b/docs/deployment/deployment_heroku.md
deleted file mode 100644
index e97163459..000000000
--- a/docs/deployment/deployment_heroku.md
+++ /dev/null
@@ -1,169 +0,0 @@
-Deploy a TOM to Heroku
-----------------------
-
-[Heroku](https://heroku.com) is a
-[PaaS](https://en.wikipedia.org/wiki/Platform_as_a_service) which allows you to
-easily deploy web applications (like a TOM) to public servers without the need for
-managing any of the underlying infrastructure yourself.
-
-Put simply: Heroku lets you make your TOM publicly available without needing to
-run servers, open firewall rules, manage domain names, etc. You simply push your
-code and Heroku will run your website for you.
-
-The service has a free tier that should be more than adequate for TOMs handling
-10s of users. However note that the free tier processing power is limited, so if
-you plan on doing lots of expensive processing on your data, you might want to
-look into alternatives.
-
-
-### Example code repository
-
-There is an example code repository,
-[TOMToolkit/herokutom](https://github.com/TOMToolkit/herokutom), which contains the
-minimal setup required to run a TOM in Heroku. It is running at
-[https://herokutom.herokuapp.com/](https://herokutom.herokuapp.com/).
-
-
-### Prerequisites
-
-1. You should have a local TOM up and running following the instructions in the
-[getting started](/introduction/getting_started) guide.
-2. You should be familiar with basic git commands.
-
-### Push your code to Github.
-This guide will use the
-[Github integration](https://devcenter.heroku.com/articles/github-integration)
-method for deploying to Heroku. This way, we can tell Heroku to redeploy your TOM
-each time we push changes to Github. Note: It's possible to [deploy to
-Heroku](https://devcenter.heroku.com/articles/git) without using Github, but
-you'll still need git.
-
-If you haven't already, push your TOM's code to Github. If you are unfamiliar with
-Git and Github, [there are many tutorials
-online](https://guides.github.com/activities/hello-world/) for getting started,
-though the specifics are beyond the scope of this tutorial.
-
-Once you have your code up on Github, you're ready to move on to the next step.
-
-### Sign up for Heroku and create an app
-
-First, start off by [signing up for a Heroku account](https://signup.heroku.com/).
-
-Once you have logged in to your account, Heroku will ask you to start a new
-project. Give it a name, but leave the pipeline stuff alone for now.
-
-After creating an app you'll be presented with a choice of Deployment methods.
-Choose Github and click the "Connect to Github" button.
-
-
-
-Once you have given Heroku access to your Github account and found your repo, your
-app should successfully be connected and your deployment dashboard should look
-like this:
-
-
-
-
-That's it for now, we'll return to this page after we've made some modifications
-to our TOM to make it work with Heroku.
-
-### Make your TOM Heroku ready
-
-There are a few additions we'll need to make to our TOM before it can run in
-Heroku. If you'd like to follow Heroku's guide directly, you can find it
-[here](https://devcenter.heroku.com/articles/django-app-configuration).
-
-#### Defining project dependencies with requirements.txt
-
-If you haven't already, define a `requirements.txt` file. This is a file which is
-used to list dependencies of your project. Heroku expects it so it knows which
-python packages it needs to install to run your app. It should look something like
-this:
-
- dataclasses
- django
- tomtoolkit
-
-Let's add 2 more lines: one for [gunicorn](https://gunicorn.org/), a high
-performance http server and
-[django-heroku](https://github.com/heroku/django-heroku) a utility that helps
-autoconfigure Django projects for Heroku. Our `requirements.txt` file should now
-look something like this:
-
- dataclasses
- django
- tomtoolkit
- gunicorn
- django-heroku
-
-You can make sure it works locally by installing your `requirements.txt`
-dependencies with `pip`:
-
- pip install -r requirements.txt
-
-#### Settings.py changes
-
-Now we need to edit our projects `settings.py` file to make it work with Heroku.
-At the top of the file, we should import django_heroku:
-
-```python
-import django_heroku
-```
-
-At the bottom of the file, we'll call a method to autoconfigure our project:
-
-```python
-django_heroku.settings(locals())
-```
-
-#### Adding a Procfile
-
-Heroku requires the presence of a `Procfile` in your project. This file tells
-Heroku how it is supposed to launch your app. Create a file `Procfile` in the root
-of your project and add these contents:
-
- release: python manage.py migrate --noinput
- web: gunicorn mytom.wsgi
-
-**Make sure to change mytom.wsgi above to the actual name of your project!**
-
-Note on the `release` command: you might want to remove this line if you'd like to
-have manual control over when your migrations are run in the future. This is
-simply a convenience for now.
-
-
-#### Push to Github and deploy
-
-Once you have made the necessary modifications to `settings.py` above, you should
-make a commit and push your code to Github.
-
-Now, navigate back to your app's dashboard on Heroku. Under the deploy tab, you
-should see a section for Manual deploy, at the bottom, with a button "Deploy
-Branch".
-
-
-
-
-Select the branch to deploy (usually "master") and click the "Deploy Branch"
-button. Heroku will begin launching your app. If all goes well, you should see
-something like this:
-
-
-
-Your TOM should now be running at https://<>.herokuapp.com.
-Congratulations!
-
-### Next steps
-
-You should spend some time familiarizing yourself with how Heroku works. As you
-may have noticed, there are many configuration options and workflows available.
-For example, just above the "Manual Deploy" section we used, there is a setting
-that allows Heroku to automatically deploy your app when you push code to Github.
-
-Also note that Heroku has limitations, especially around storing data on disk. By
-default, **Heroku only keeps files on disk for a maximum of 24 hours**. If you
-plan on storing data (such as fits files or other supplementary data) you will
-have to use an external stoage service. In this case, you might want to read ahead
-on how to [Use Amazon S3 to Store Data for a
-TOM](https://tomtoolkit.github.io/docs/amazons3).
-
diff --git a/docs/deployment/deployment_heroku.rst b/docs/deployment/deployment_heroku.rst
new file mode 100644
index 000000000..ed5e4f748
--- /dev/null
+++ b/docs/deployment/deployment_heroku.rst
@@ -0,0 +1,195 @@
+Deploy a TOM to Heroku
+----------------------
+
+`Heroku `__ is a
+`PaaS `__ which
+allows you to easily deploy web applications (like a TOM) to public
+servers without the need for managing any of the underlying
+infrastructure yourself.
+
+Put simply: Heroku lets you make your TOM publicly available without
+needing to run servers, open firewall rules, manage domain names, etc.
+You simply push your code and Heroku will run your website for you.
+
+The service has a free tier that should be more than adequate for TOMs
+handling 10s of users. However note that the free tier processing power
+is limited, so if you plan on doing lots of expensive processing on your
+data, you might want to look into alternatives.
+
+Example code repository
+~~~~~~~~~~~~~~~~~~~~~~~
+
+There is an example code repository,
+`TOMToolkit/herokutom `__,
+which contains the minimal setup required to run a TOM in Heroku. It is
+running at https://herokutom.herokuapp.com/.
+
+Prerequisites
+~~~~~~~~~~~~~
+
+1. You should have a local TOM up and running following the instructions
+ in the `getting started `__ guide.
+2. You should be familiar with basic git commands.
+
+Push your code to Github.
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+This guide will use the `Github
+integration `__
+method for deploying to Heroku. This way, we can tell Heroku to redeploy
+your TOM each time we push changes to Github. Note: It’s possible to
+`deploy to Heroku `__ without
+using Github, but you’ll still need git.
+
+If you haven’t already, push your TOM’s code to Github. If you are
+unfamiliar with Git and Github, `there are many tutorials
+online `__ for
+getting started, though the specifics are beyond the scope of this
+tutorial.
+
+Once you have your code up on Github, you’re ready to move on to the
+next step.
+
+Sign up for Heroku and create an app
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+First, start off by `signing up for a Heroku
+account `__.
+
+Once you have logged in to your account, Heroku will ask you to start a
+new project. Give it a name, but leave the pipeline stuff alone for now.
+
+After creating an app you’ll be presented with a choice of Deployment
+methods. Choose Github and click the “Connect to Github” button.
+
+|image0|
+
+Once you have given Heroku access to your Github account and found your
+repo, your app should successfully be connected and your deployment
+dashboard should look like this:
+
+|image1|
+
+That’s it for now, we’ll return to this page after we’ve made some
+modifications to our TOM to make it work with Heroku.
+
+Make your TOM Heroku ready
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+There are a few additions we’ll need to make to our TOM before it can
+run in Heroku. If you’d like to follow Heroku’s guide directly, you can
+find it
+`here `__.
+
+Defining project dependencies with requirements.txt
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+If you haven’t already, define a ``requirements.txt`` file. This is a
+file which is used to list dependencies of your project. Heroku expects
+it so it knows which python packages it needs to install to run your
+app. It should look something like this:
+
+::
+
+ dataclasses
+ django
+ tomtoolkit
+
+Let’s add 2 more lines: one for `gunicorn `__, a
+high performance http server and
+`django-heroku `__ a utility
+that helps autoconfigure Django projects for Heroku. Our
+``requirements.txt`` file should now look something like this:
+
+::
+
+ dataclasses
+ django
+ tomtoolkit
+ gunicorn
+ django-heroku
+
+You can make sure it works locally by installing your
+``requirements.txt`` dependencies with ``pip``:
+
+::
+
+ pip install -r requirements.txt
+
+Settings.py changes
+^^^^^^^^^^^^^^^^^^^
+
+Now we need to edit our projects ``settings.py`` file to make it work
+with Heroku. At the top of the file, we should import django_heroku:
+
+.. code:: python
+
+ import django_heroku
+
+At the bottom of the file, we’ll call a method to autoconfigure our
+project:
+
+.. code:: python
+
+ django_heroku.settings(locals())
+
+Adding a Procfile
+^^^^^^^^^^^^^^^^^
+
+Heroku requires the presence of a ``Procfile`` in your project. This
+file tells Heroku how it is supposed to launch your app. Create a file
+``Procfile`` in the root of your project and add these contents:
+
+::
+
+ release: python manage.py migrate --noinput
+ web: gunicorn mytom.wsgi
+
+**Make sure to change mytom.wsgi above to the actual name of your
+project!**
+
+Note on the ``release`` command: you might want to remove this line if
+you’d like to have manual control over when your migrations are run in
+the future. This is simply a convenience for now.
+
+Push to Github and deploy
+^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Once you have made the necessary modifications to ``settings.py`` above,
+you should make a commit and push your code to Github.
+
+Now, navigate back to your app’s dashboard on Heroku. Under the deploy
+tab, you should see a section for Manual deploy, at the bottom, with a
+button “Deploy Branch”.
+
+|image2|
+
+Select the branch to deploy (usually “master”) and click the “Deploy
+Branch” button. Heroku will begin launching your app. If all goes well,
+you should see something like this:
+
+|image3|
+
+Your TOM should now be running at https://<>.herokuapp.com.
+Congratulations!
+
+Next steps
+~~~~~~~~~~
+
+You should spend some time familiarizing yourself with how Heroku works.
+As you may have noticed, there are many configuration options and
+workflows available. For example, just above the “Manual Deploy” section
+we used, there is a setting that allows Heroku to automatically deploy
+your app when you push code to Github.
+
+Also note that Heroku has limitations, especially around storing data on
+disk. By default, **Heroku only keeps files on disk for a maximum of 24
+hours**. If you plan on storing data (such as fits files or other
+supplementary data) you will have to use an external stoage service. In
+this case, you might want to read ahead on how to `Use Amazon S3 to
+Store Data for a TOM `__.
+
+.. |image0| image:: /_static/heroku_deploy_doc/githubintegration.png
+.. |image1| image:: /_static/heroku_deploy_doc/githubconnected.png
+.. |image2| image:: /_static/heroku_deploy_doc/herokudeploybranch.png
+.. |image3| image:: /_static/heroku_deploy_doc/branchdeployed.png
diff --git a/docs/deployment/deployment_tips.md b/docs/deployment/deployment_tips.md
deleted file mode 100644
index a5030e1bc..000000000
--- a/docs/deployment/deployment_tips.md
+++ /dev/null
@@ -1,42 +0,0 @@
-General Deployment Tips
----
-
-When it comes to deploying your tom for general use, there are a few things you
-might want to consider.
-
-### Choosing a database
-By default Django (and thus TOMs) use Sqlite as their database backend. Sqlite is
-sufficient for the majority of use cases and can scale up to the millions if not
-billions of rows, as long as you have the disk space.
-
-The one place where Sqlite falls behind other databases is it's performance under
-heavy concurrent writes. So if you are writing a TOM that, for example, listens
-to the ZTF, LSST, and SCOUT alert streams and creates targets from each alert
-you might want to look into Postgresql or MySQL.
-
-
-### Set your TOM's hostname in the default site
-In your TOM's admin area (/admin/) on your production TOM
-you will notice a section called "Sites." There
-should be one site object, you should edit it so that its hostname is accurate for
-production. Some functionalities of the TOM rely on this value to properly set up
-redirects, etc.
-
-
-### Basic security
-If you are exposing your TOM to the internet you should make sure that basic
-security precautions have been taken. Make sure that any views which expose
-sensitive data, perform any kind of modification to the database or cause large
-amounts of server load are properly protected and require authentication.
-
-If you plan on making your TOM open source, take care not to check in any secrets,
-passwords, or credentials. This includes database settings, API keys, or a
-multitude of other things that you wouldn't want to throw out on the internet for
-everyone to see. Note that if you are using git, removing a secret from a file and
-then committing it **does not remove the secret** it will still exist in the
-repo's history and be trivially accessible. You will need to clean your repo's
-history if you commit and sensitive data.
-
-Enforce basic password requirements (TOMs by default will do this) and encourage
-your users to exercise basic security measures, like using a password manager and
-not reusing passwords.
diff --git a/docs/deployment/deployment_tips.rst b/docs/deployment/deployment_tips.rst
new file mode 100644
index 000000000..0632adf0d
--- /dev/null
+++ b/docs/deployment/deployment_tips.rst
@@ -0,0 +1,50 @@
+General Deployment Tips
+-----------------------
+
+When it comes to deploying your tom for general use, there are a few
+things you might want to consider.
+
+Choosing a database
+~~~~~~~~~~~~~~~~~~~
+
+By default Django (and thus TOMs) use Sqlite as their database backend.
+Sqlite is sufficient for the majority of use cases and can scale up to
+the millions if not billions of rows, as long as you have the disk
+space.
+
+The one place where Sqlite falls behind other databases is it’s
+performance under heavy concurrent writes. So if you are writing a TOM
+that, for example, listens to the ZTF, LSST, and SCOUT alert streams and
+creates targets from each alert you might want to look into Postgresql
+or MySQL.
+
+Set your TOM’s hostname in the default site
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+In your TOM’s admin area (/admin/) on your production TOM you will
+notice a section called “Sites.” There should be one site object, you
+should edit it so that its hostname is accurate for production. Some
+functionalities of the TOM rely on this value to properly set up
+redirects, etc.
+
+Basic security
+~~~~~~~~~~~~~~
+
+If you are exposing your TOM to the internet you should make sure that
+basic security precautions have been taken. Make sure that any views
+which expose sensitive data, perform any kind of modification to the
+database or cause large amounts of server load are properly protected
+and require authentication.
+
+If you plan on making your TOM open source, take care not to check in
+any secrets, passwords, or credentials. This includes database settings,
+API keys, or a multitude of other things that you wouldn’t want to throw
+out on the internet for everyone to see. Note that if you are using git,
+removing a secret from a file and then committing it **does not remove
+the secret** it will still exist in the repo’s history and be trivially
+accessible. You will need to clean your repo’s history if you commit and
+sensitive data.
+
+Enforce basic password requirements (TOMs by default will do this) and
+encourage your users to exercise basic security measures, like using a
+password manager and not reusing passwords.
diff --git a/docs/deployment/index.rst b/docs/deployment/index.rst
index 3d7ea89af..8b7e6f5fb 100644
--- a/docs/deployment/index.rst
+++ b/docs/deployment/index.rst
@@ -1,8 +1,8 @@
-Deployment
-==========
+Deploying your TOM Online
+=========================
.. toctree::
- :maxdepth: 1
+ :maxdepth: 2
:hidden:
deployment_tips
diff --git a/docs/examples.md b/docs/examples.md
deleted file mode 100644
index 4eb7e1997..000000000
--- a/docs/examples.md
+++ /dev/null
@@ -1,26 +0,0 @@
-Example TOMs
----
-
-### SNEx
-
-The [Supernova Exchange](https://supernova.exchange/public/) is an interface for viewing and sharing observational data of supernovae, and for requesting and managing observations with the Las Cumbres Observatory network. In order to make it more maintainable, it is being rewritten from scratch using the TOM Toolkit, which has already resulted in orders of magnitude fewer lines of code. The code can be found and referenced on [Github](https://github.com/jfrostburke/snex2/).
-
-### Asteroid Tracker
-
-[Asteroid Tracker](https://asteroidtracker.lco.global/) is an educational TOM built for Asteroid Day. It allows students and teachers to submit one-click observations of specific asteroids and see the resulting images. Originally built from scratch, it's being rewritten using the TOM Toolkit, which will allow the underlying TOM to be used with multiple front-ends for completely different educational purposes.
-
-### Microlensing TOM
-
-The [Microlensing TOM](https://github.com/KSNikolaus/ZTF_TOM) is being written in order to identify microlensing events from ZTF and conduct follow-up observations.
-
-### PhotTOM
-
-The [ROME/REA TOM](https://github.com/rachel3834/romerea_phot_tom) is being built to manage ROME/REA photometry for the [LCO key project of the same name](https://robonet.lco.global/).
-
-### Calibration TOM
-
-LCO is rewriting an existing piece of software that automatically schedules nightly telescope calibrations using the TOM Toolkit called the [Calibration TOM](https://github.com/LCOGT/calibration-tom/).
-
-### Others
-
-There are a few other TOMs in development that we're aware of, but if you're developing a TOM, feel free to contribute to this page, or let us know and we'll take care of it for you.
\ No newline at end of file
diff --git a/docs/examples.rst b/docs/examples.rst
new file mode 100644
index 000000000..95fbfef70
--- /dev/null
+++ b/docs/examples.rst
@@ -0,0 +1,66 @@
+Example TOMs
+------------
+
+SNEx
+~~~~
+
+The `Supernova Exchange `__ is an
+interface for viewing and sharing observational data of supernovae, and
+for requesting and managing observations with the Las Cumbres
+Observatory network. In order to make it more maintainable, it is being
+rewritten from scratch using the TOM Toolkit, which has already resulted
+in orders of magnitude fewer lines of code. The code can be found and
+referenced on `Github `__.
+
+Asteroid Tracker
+~~~~~~~~~~~~~~~~
+
+`Asteroid Tracker `__ is an
+educational TOM built for Asteroid Day. It allows students and teachers
+to submit one-click observations of specific asteroids and see the
+resulting images. Originally built from scratch, it’s being rewritten
+using the TOM Toolkit, which will allow the underlying TOM to be used
+with multiple front-ends for completely different educational purposes.
+
+Microlensing TOM (MOP)
+~~~~~~~~~~~~~~~~~~~~~~
+
+The `Microlensing Observing Platform `__ is the core interface of the OMEGA Key Project. It is designed to harvest and prioritize microlensing events from various surveys, then submit additional observations with the Las Cumbres Observatory telescopes automatically.
+
+
+PhotTOM
+~~~~~~~
+
+The `ROME/REA TOM `__ is
+being built to manage ROME/REA photometry for the `LCO key project of
+the same name `__.
+
+Calibration TOM
+~~~~~~~~~~~~~~~
+
+LCO is rewriting an existing piece of software that automatically
+schedules nightly telescope calibrations using the TOM Toolkit called
+the `Calibration TOM `__.
+
+PANOPTES TOM
+~~~~~~~~~~~~
+
+The `PANOPTES TOM `__ is being
+built to enable their community to coordinate observations for the
+`PANOPTES citizen science project `__, which
+aims to detect transiting exoplanets.
+
+
+AMON TOM
+~~~~~~~~
+
+Black Hole TOM
+~~~~~~~~~~~~~~
+Black Hole TOM (BHTOM) aims at coordinating the photometric and spectroscopic follow-up observations of targets requiring long-term monitoring. This includes long lasting microlensing events reported by Gaia and other surveys, likely caused by galactic black holes. The system lists targets according to their priorities and allows for triggering robotic observations. It also allows users of any partner observatory to submit their raw photometric and spectroscopic data, which gets automatically processed and calibrated. BHTOM is developed as part of the Time Domain Astronony work package of the European OPTICON grant by the team at the University of Warsaw, Poland, with support from LCO. Website: http://visata.astrouw.edu.pl:8080
+
+Others
+~~~~~~
+
+There are a few other TOMs in development that we’re aware of, but if
+you’re developing a TOM, feel free to contribute to this page, or let us
+know and we’ll take care of it for you.
diff --git a/docs/index.rst b/docs/index.rst
index 8a07b681e..445926f36 100644
--- a/docs/index.rst
+++ b/docs/index.rst
@@ -6,115 +6,85 @@ Welcome to the TOM Toolkit's documentation!
:hidden:
introduction/index
- customization/index
- advanced/index
- deployment/index
- introduction/faqs
+ introduction/about
+ introduction/support
+ introduction/troubleshooting
+
Introduction
------------
-The TOM (Target and Observation Manager) Toolkit project was started in early 2018 with the goal of simplifying the development of next generation software for the rapidly evolving field of astronomy. Read more :doc:`about TOMs` and the motivation for them.
-
-Interested in seeing what a TOM can do? Take a look at our `demonstration TOM `_, where we show off the features of the TOM Toolkit.
-
-Are you looking to run a TOM of your own? This documentation is a good place to get started. The source code for the project is also available on Github.
+The TOM (Target and Observation Manager) Toolkit project was started in early 2018 with the goal of simplifying the development of next generation software for the rapidly evolving field of astronomy. Read more :doc:`about TOMs` and the motivation for them.
-Start with the :doc:`introduction` if you are new to using the TOM Toolkit.
-
-If you'd like to know what we're working on, check out the `TOM Toolkit project board `_.
-
-:doc:`Architecture ` - This document describes the architecture of the TOM Toolkit at a
+:doc:`TOM Toolkit Architecture ` - This document describes the architecture of the TOM Toolkit at a
high level. Read this first if you're interested in how the TOM Toolkit works.
-:doc:`Getting Started ` - First steps for getting a TOM up and running.
+:doc:`Getting Started with the TOM Toolkit` - First steps for getting a TOM up and running.
-:doc:`Workflow ` - The general workflow used with TOMs.
+:doc:`TOM Workflow ` - The general workflow used with TOMs.
:doc:`Programming Resources ` - Resources for learning the core components of the TOM Toolkit:
HTML, CSS, Python, and Django
:doc:`Frequently Asked Questions ` - Look here for a potential quick answer to a common question.
-Extending and Customizing
--------------------------
-
-Start here to learn how to customize the look and feel of your TOM or add new functionality.
-
-:doc:`Custom Settings ` - Settings available to the TOM Toolkit which you may want to
-configure.
+:doc:`Troubleshooting ` - Find solutions to common problems or information on how to debug an issue.
-:doc:`Customizing TOM Templates ` - Learn how to override built in TOM templates to
-change the look and feel of your TOM.
-
-:doc:`Adding new Pages to your TOM ` - Learn how to add entirely new pages to your TOM,
-displaying static html pages or dynamic database-driven content.
-
-:doc:`Adding Custom Target Fields ` - Learn how to add custom fields to your TOM Targets if the
-defaults do not suffice.
-
-:doc:`Adding Custom Data Processing ` - Learn how you can process data into your
-TOM from uploaded data products.
-
-:doc:`Building a TOM Alert Broker ` - Learn how to build an Alert Broker module to add new
-sources of targets to your TOM.
+Interested in seeing what a TOM can do? Take a look at our `demonstration TOM `_, where we show off the features of the TOM Toolkit.
-:doc:`Changing Request Submission Behavior ` - Learn how to customize the LCO
-Observation Module in order to add additional parameters to observation requests sent to the LCO Network.
+If you'd like to know what we're working on, check out the `TOM Toolkit project board `_.
-:doc:`Creating Plots from TOM Data ` - Learn how to create plots using plot.ly and your TOM
-data to display anywhere in your TOM.
-:doc:`The Permissions System ` - Use the permissions system to limit access to targets in your
-TOM.
+Topics
+------
-:doc:`Automating Tasks ` - Run commands automatically to keep your TOM working even when you
-aren’t
+.. toctree::
+ :maxdepth: 2
+ :hidden:
-Advanced Topics
----------------
+ targets/index
+ brokers/index
+ observing/index
+ managing_data/index
+ customization/index
+ common/permissions
+ common/latex_generation
+ code/index
+ deployment/index
+ common/customsettings
-:doc:`Background Tasks ` - Learn how to set up an asynchronous task library to handle long
-running and/or concurrent functions.
-:doc:`Building a TOM Observation Facility Module ` - Learn to build a module which will
-allow your TOM to submit observation requests to observatories.
+:doc:`Targets ` - Learn all about how to manage Targets in a TOM.
-:doc:`Running Custom Code Hooks ` - Learn how to run your own scripts when certain actions happen
-within your TOM (for example, an observation completes).
+:doc:`Brokers ` - Find out about querying brokers in the TOM, which are available, and writing your own.
-:doc:`Scripting your TOM with Jupyter Notebooks ` - Use a Jupyter notebook (or just a python
-console/scripts) to interact directly with your TOM.
+:doc:`Observing ` - Tutorials on submitting observations, customizing submission, and the available facilities.
-:doc:`Observing and cadence strategies ` - Learn about observing and cadence strategies and how to write a
-custom cadence strategy to automate a series of observations.
+:doc:`Managing Data ` - Customize plots, upload data, and even integrate a data reduction pipeline.
-:doc:`LaTeX table generation ` - Learn how to generate LaTeX for certain models and add LaTeX
-generators for other models.
+:doc:`Customization ` - Customize and create new views in your TOM.
-Deployment
-----------
+:doc:`The Permissions System ` - Use the permissions system to limit access to targets in your TOM.
-Once you’ve got a TOM up and running on your machine, you’ll probably want to deploy it somewhere so it is permanently
-accessible by you and your colleagues.
+:doc:`LaTeX Generation ` - Generate data tables for your targets and observations
-:doc:`General Deployment Tips ` - Read this first before deploying your TOM for others to use.
+:doc:`Interacting with your TOM through code ` - Learn how to programmatically interact with your TOM.
-:doc:`Deploy to Heroku ` - Heroku is a PaaS that allows you to publicly deploy your web applications without the need for managing the infrastructure yourself.
+:doc:`Deploying your TOM Online ` - Resources for deploying your TOM to a cloud provider
-:doc:`Using Amazon S3 to Store Data for a TOM ` - Enable storing data on the cloud storage service Amazon S3 instead of your local disk.
+:doc:`TOM Settings ` - Reference and description for the available settings values to be added to/edited in your project's ``settings.py``.
Contributing
------------
If you find an issue, you need help with your TOM, you have a useful idea, or you wrote a module you'd like to be
-included in the TOM Toolkit, start with the :doc:`Contribution Guide `.
+included in the TOM Toolkit, start with the :doc:`Contribution Guide `.
Support
-------
Looking for help? Want to request a feature? Have questions about Github Issues? Take a look at the :doc:`support guide
-`.
+`.
If you just need an idea, checkout out the :doc:`examples` of existing TOMs built with the TOM Toolkit.
@@ -122,10 +92,9 @@ If you just need an idea, checkout out the :doc:`examples` of existing
:maxdepth: 1
:hidden:
- contributing
- support
examples
- about
+ introduction/contributing
+ Release Notes
Github
API Documentation
@@ -152,4 +121,4 @@ About the TOM Toolkit
The TOM Toolkit is managed by Las Cumbres Observatory, with generous financial support from the `Heising-Simons Foundation `_ and the `Zegar Family Foundation `_.
-Read about the project and the motivations behind it on the :doc:`About page `.
+Read about the project and the motivations behind it on the :doc:`About page `.
diff --git a/docs/introduction/about.rst b/docs/introduction/about.rst
new file mode 100644
index 000000000..01de8d295
--- /dev/null
+++ b/docs/introduction/about.rst
@@ -0,0 +1,72 @@
+About the TOM Toolkit
+---------------------
+
+What’s a TOM?
+~~~~~~~~~~~~~
+
+It stands for Target and Observation Manager, and its a software package
+designed to facilitate astronomical observing projects and
+collaborations.
+
+Though useful for a wide range of projects, TOM systems are particularly
+important for programs with a large number of potential targets and/or
+observations.
+
+TOM systems perform some or all of these functions:
+
+- Harvest target alerts, or upload catalogs of targets of interest to
+ the project science goals.
+- Search and cross-match additional information on targets from
+ catalogs and archives.
+- Store information from the project’s own analysis of the targets,
+ related data and observations.
+- Provide informative displays of the targets, data and observing
+ program.
+- Provide flexible search capabilities on parameters that are relevant
+ to the science.
+- Provide tools to plan appropriate observations.
+- Enable observations to be requested from telescope facilities.
+- Receive information about the status of observation requests.
+- Harvest data obtained as a result of their observing requests.
+- Facilitate the sharing of information and data.
+
+Motivation for a TOM Toolkit
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Many projects, from several branches of astronomy, have found it
+necessary to develop TOM systems. Current examples include the PTF
+Marshall and NASA’s ExoFOP, as well as those customized for the LCO
+Network: SNEx, NEO Exchange and RoboNet. These tools provide
+capabilities which enable the projects to identify and evaluate high
+priority targets in good time to plan and conduct suitable observations,
+and to analyze the results. These capabilities have proven to be
+essential for existing projects to keep track of their observing program
+and to achieve their scientific goals. They are likely to become
+increasingly vital as next generation surveys produce ever-larger and
+more rapidly-evolving target lists.
+
+However, designing the existing TOM systems required high levels of
+expertise in database and software development that are not common among
+astronomers.
+
+No two TOM systems are identical, as astronomers strongly prefer to
+directly control the science-specific aspects of their projects such as
+target selection, observation templates and analysis techniques. At the
+same time, while all of these systems are customized for the science
+goals of the projects they support, much of their underlying
+infrastructure and functions are very similar.
+
+What’s needed is a software package that lets astronomers easily build a
+TOM, customized to suit the needs of their project, without becoming an
+IT expert or software engineer.
+
+Financial Support
+~~~~~~~~~~~~~~~~~
+
+The TOM Toolkit has been made possible through generous financial
+support from the `Heising-Simons
+Foundation `_ and the `Zegar Family
+Foundation `_.
+
+.. image:: /_static/hs.jpg
+.. image:: /_static/zff.png
diff --git a/docs/introduction/contributing.rst b/docs/introduction/contributing.rst
new file mode 100644
index 000000000..a44bcecbf
--- /dev/null
+++ b/docs/introduction/contributing.rst
@@ -0,0 +1,116 @@
+Contributing
+------------
+
+This page will go over the process for contributing to the TOM Toolkit.
+
+Contributing Code/Documentation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you’re interested in contributing code to the project, thank you! For
+those unfamiliar with the process of contributing to an open-source
+project, you may want to read through Github’s own short informational
+section on `how to submit a
+contribution `__.
+
+Identifying a starting point
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The best place to begin contributing is by first looking at the `Github
+issues page `__, to see
+what’s currently needed. Issues that don’t require much familiarity with
+the TOM Toolkit will be tagged appropriately.
+
+Familiarizing yourself with Git
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you are not familiar with git, we encourage you to briefly look at
+the `Git
+Basics `__
+page.
+
+Git Workflow
+~~~~~~~~~~~~
+
+The workflow for submitting a code change is, more or less, the
+following:
+
+1. Fork the TOM Toolkit repository to your own Github account. |image0|
+2. Clone the forked repository to your local working machine.
+
+::
+
+ git clone git@github.com:/tom_base.git
+
+3. Add the original “upstream” repository as a remote.
+
+::
+
+ git remote add upstream https://github.com/TOMToolkit/tom_base.git
+
+4. Ensure that you’re synchronizing your repository with the “upstream”
+ one relatively frequently.
+
+::
+
+ git fetch upstream
+ git merge upstream/main
+
+5. Create and checkout a branch for your changes (see `Branch
+ Naming <#branch-naming>`__).
+
+::
+
+ git checkout -b
+
+6. Commit frequently, and push your changes to Github. Be sure to merge
+ main in before submitting your pull request.
+
+::
+
+ git push origin
+
+7. When your code is complete and tested, create a pull request from the
+ upstream TOM Toolkit repository. |image1|
+
+8. Be sure to click “compare across forks” in order to see your branch!
+ |image2|
+
+9. We may ask for some updates to your pull request, so revise as
+ necessary and push when revisions are complete. This will
+ automatically update your pull request.
+
+Branch Naming
+~~~~~~~~~~~~~
+
+Branch names should be prefixed with the purpose of the branch, be it a
+bugfix or an enhancement, along with a descriptive title for the branch.
+
+::
+
+ bugfix/fix-typo-target-detail
+ feature/reticulating-splines
+ enhancement/refactor-planning-tool
+
+Code Style
+~~~~~~~~~~
+
+We recommend that you use a linter, as all pull requests must pass a
+``pycodestyle`` check. We also recommend configuring your editor to
+automatically remove trailing whitespace, add newlines on save, and
+other such helpful style corrections. You can check if your styling will
+meet standards before submitting a pull request by doing a
+``pip install pycodestyle`` and running the same command our Travis
+build does:
+
+::
+
+ pycodestyle tom_* --exclude=*/migrations/* --max-line-length=120
+
+Documentation
+~~~~~~~~~~~~~
+
+We require any new features to
+
+.. |image0| image:: /_static/fork.png
+.. |image1| image:: /_static/pull-request.png
+.. |image2| image:: /_static/compare-across-forks.png
diff --git a/docs/introduction/faqs.md b/docs/introduction/faqs.md
deleted file mode 100644
index e0dc4e22d..000000000
--- a/docs/introduction/faqs.md
+++ /dev/null
@@ -1,82 +0,0 @@
-FAQ
----
-
-### Can I use Jupyter Notebooks with my TOM?
-
-Yes. First install jupyterlab into your TOM virtualenv:
-
- pip install jupyterlab
-
-Inside your TOM directory, use the following management command to launch the
-notebook server:
-
- ./manage.py shell_plus --notebook
-
-Under the new notebook menu, choose "Django Shell-Plus". This will create a new
-notebook in the correct TOM context.
-
-There is also a [tutorial](/advanced/scripts) on interacting with your TOM using
-Jupyter notebooks.
-
-### What are tags on the Target form?
-You can add tags to targets via the target create/update forms or
-programmatically. These are meant to be arbitrary data associated with a target.
-You can then search for targets via tags on the target list page, by entering the
-"key" and/or "value" fields in the filter list. They will also be displayed on the
-target detail pages.
-
-If you'd like to have more control over extra target data, see the documentation
-on [Adding Custom Target Fields](/customization/target_fields).
-
-### I try to observe a target with LCO but get an error.
-
-You might not have added your LCO api key to your settings file under the
-`FACILITIES` settings. See [Custom Settings](/customization/customsettings#facilities) for
-more details.
-
-### How do I create a super user (PI)?
-You can create a new superuser using the built in management command:
-
- ./manage.py createsuperuser
-
-The `manage.py` file can be found in the root of your project.
-
-Alternatively, you can give a user superuser status if you are already logged
-in as a superuser by visiting the admin page for users:
-[http://127.0.0.1/admin/auth/user/](http://127.0.0.1/admin/auth/user/)
-
-
-### My science requires more parameters than are provided by the TOM Toolkit.
-It is possible to add additional parameters to your targets within the TOM. See
-the documentation on [Adding Custom Target Fields](/customization/target_fields).
-
-
-### Yuck! My TOM is ugly. How do I change how it looks?
-You have a few options. If you'd like to rearrange the layout or information on
-the page, you can follow the tutorial on
-[Customizing your TOM](/customization/customize_templates). If you'd like to modify colors,
-typography, etc you'll want to use CSS.
-[W3Schools](https://www.w3schools.com/Css/) is a good resource if you are
-unfamiliar with Cascading Style Sheets.
-
-
-### How do I add a new page to my TOM?
-We would recommend you read the [Django tutorial](https://docs.djangoproject.com/en/2.2/contents/)
-🙂. But if you want the quick and dirty, edit the `urls.py` (located next to
-`settings.py`):
-
-```python
-from django.urls import path, include
-from django.views.generic import TemplateView
-
-urlpatterns = [
- path('', include('tom_common.urls')),
- path('newpage/', TemplateView.as_view(template_name='newpage.html'), name='newpage')
-]
-```
-
-And make sure `newpage.html` is located within the `templates/` directory in your
-project.
-
-This will make the contents of `newpage.html` available under the path
-[/newpage/](http://127.0.0.1/newpage/).
diff --git a/docs/introduction/faqs.rst b/docs/introduction/faqs.rst
new file mode 100644
index 000000000..0485a788d
--- /dev/null
+++ b/docs/introduction/faqs.rst
@@ -0,0 +1,121 @@
+FAQ
+###
+
+Can I use Jupyter Notebooks with my TOM?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Yes. First install jupyterlab into your TOM virtualenv:
+
+::
+
+ pip install jupyterlab
+
+Inside your TOM directory, use the following management command to
+launch the notebook server:
+
+::
+
+ ./manage.py shell_plus --notebook
+
+Under the new notebook menu, choose “Django Shell-Plus”. This will
+create a new notebook in the correct TOM context.
+
+There is also a `tutorial <../common/scripts>`__ on interacting with
+your TOM using Jupyter notebooks.
+
+What are tags on the Target form?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can add tags to targets via the target create/update forms or
+programmatically. These are meant to be arbitrary data associated with a
+target. You can then search for targets via tags on the target list
+page, by entering the “key” and/or “value” fields in the filter list.
+They will also be displayed on the target detail pages.
+
+If you’d like to have more control over extra target data, see the
+documentation on `Adding Custom Target
+Fields <../targets/target_fields>`__.
+
+I try to observe a target with LCO but get an error.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You might not have added your LCO api key to your settings file under
+the ``FACILITIES`` settings. See `Custom
+Settings <../uncategorized/customsettings#facilities>`__ for more
+details.
+
+How do I create a super user (PI)?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You can create a new superuser using the built in management command:
+
+::
+
+ ./manage.py createsuperuser
+
+The ``manage.py`` file can be found in the root of your project.
+
+Alternatively, you can give a user superuser status if you are already
+logged in as a superuser by visiting the admin page for users:
+http://127.0.0.1/admin/auth/user/
+
+My science requires more parameters than are provided by the TOM Toolkit.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+It is possible to add additional parameters to your targets within the
+TOM. See the documentation on `Adding Custom Target
+Fields <../targets/target_fields>`__.
+
+Yuck! My TOM is ugly. How do I change how it looks?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You have a few options. If you’d like to rearrange the layout or
+information on the page, you can follow the tutorial on `Customizing
+your TOM <../customization/customize_templates>`__. If you’d like to
+modify colors, typography, etc you’ll want to use CSS.
+`W3Schools `__ is a good resource if you
+are unfamiliar with Cascading Style Sheets.
+
+How do I add a new page to my TOM?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We would recommend you read the `Django
+tutorial `__ 🙂. But if
+you want the quick and dirty, edit the ``urls.py`` (located next to
+``settings.py``):
+
+.. code:: python
+
+ from django.urls import path, include
+ from django.views.generic import TemplateView
+
+ urlpatterns = [
+ path('', include('tom_common.urls')),
+ path('newpage/', TemplateView.as_view(template_name='newpage.html'), name='newpage')
+ ]
+
+And make sure ``newpage.html`` is located within the ``templates/``
+directory in your project.
+
+This will make the contents of ``newpage.html`` available under the path
+`/newpage/ `__.
+
+Who is AnonymousUser?
+~~~~~~~~~~~~~~~~~~~~~
+
+AnonymousUser is a special profile that django-guardian, our permissions
+library, creates automatically. AnonymousUser represents an
+unauthenticated user. The user has no first name, last name, or
+password, and allows unauthenticated users to view unprotected pages
+within your TOM. You can choose to delete the user if you don’t want any
+pages to be visible without logging in.
+
+How can I display an error message when authentication to an external facility fails?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+For any modules exposing external services, such as brokers, harvesters,
+or facilities, a failed authentication should raise an
+``ImproperCredentialsException``. Exceptions of this type are caught by
+the TOM Toolkit’s built-in ``ExternalServiceMiddleware``. This
+middleware will display an error at the top of the page and redirect the
+user to the home page.
diff --git a/docs/introduction/getting_started.md b/docs/introduction/getting_started.md
deleted file mode 100644
index 593563c5b..000000000
--- a/docs/introduction/getting_started.md
+++ /dev/null
@@ -1,91 +0,0 @@
-Getting Started with the TOM Toolkit
-------------------------------------
-
-So you've decided to run a Target Observation Manager. This article will help you get started.
-
-The TOM Toolkit is a [Django](https://www.djangoproject.com/) project. This means you'll be running
-an application based on the Django framework when you run a TOM. If you decide to customize
-your TOM, you'll be working in Django. You'll likely need some basic understanding of python
-and we recommend all users work their way through the
-[Django tutorial](https://docs.djangoproject.com/en/2.1/contents/) first before starting with
-the TOM Toolkit. It doesn't take long, and you most likely won't need to utilize any advanced
-features.
-
-Ready to go? Let's get started.
-
-### Prerequisites
-
-The TOM toolkit requires Python >= 3.7
-
-If you are using Python 3.6 and cannot upgrade to 3.7, install the `dataclasses`
-backport:
-
- pip install dataclasses
-
-### Installing the TOM Toolkit and Django
-
-First, we recommend using a
-[virtual environment](https://docs.python.org/3/tutorial/venv.html) for your
-project.
-This will keep your TOM python packages seperate from your system python packages.
-
- python3 -m venv tom_env/
-
-Now that we have created the virtual environment, we can activate it:
-
- source tom_env/bin/activate
-
-You should now see `(tom_env)` prepended to your terminal prompt.
-
-Now, install the TOM Toolkit:
-
- pip install tomtoolkit
-
-...and create a new project, just like in the tutorial:
-
- django-admin startproject mytom
-
-You should now have a fully functional standard Django installation inside the
-`mytom` folder, with the TOM dependencies installed as well.
-
-### Getting started with the `tom_setup` script.
-
-We need to add the `tom_setup` app to our project's `INSTALLED_APPS`. Locate the
-`settings.py` file inside your project directory (usually in a subdirectory of the
-main folder, i.e. mytom/mytom/settings.py) and edit it so that it looks like this:
-
-```python
-INSTALLED_APPS = [
- 'django.contrib.admin',
- 'django.contrib.auth',
- 'django.contrib.contenttypes',
- 'django.contrib.sessions',
- 'django.contrib.messages',
- 'django.contrib.staticfiles',
- 'tom_setup',
-]
-```
-
-### Run the setup script
-
-The `tom_setup` app contains a script that will bootstrap a new TOM in your
-current project. Run it:
-
- ./manage.py tom_setup
-
-The install script will ask you a few questions and then install your TOM.
-
-### Running the dev server
-
-Now that the toolkit is installed, you're ready to try it out!
-
-First, run the necessary migrations:
-
- ./manage.py migrate
-
-Now, start the dev server:
-
- ./manage.py runserver
-
-Your new TOM should now be running on [http://127.0.0.1:8000](http://127.0.0.1:8000)!
-
diff --git a/docs/introduction/getting_started.rst b/docs/introduction/getting_started.rst
new file mode 100644
index 000000000..3f3a8349a
--- /dev/null
+++ b/docs/introduction/getting_started.rst
@@ -0,0 +1,116 @@
+Getting Started with the TOM Toolkit
+------------------------------------
+
+So you’ve decided to run a Target Observation Manager. This article will
+help you get started.
+
+The TOM Toolkit is a `Django `__
+project. This means you’ll be running an application based on the Django
+framework when you run a TOM. If you decide to customize your TOM,
+you’ll be working in Django. You’ll likely need some basic understanding
+of python and we recommend all users work their way through the `Django
+tutorial `__ first
+before starting with the TOM Toolkit. It doesn’t take long, and you most
+likely won’t need to utilize any advanced features.
+
+Ready to go? Let’s get started.
+
+Prerequisites
+~~~~~~~~~~~~~
+
+The TOM toolkit requires Python >= 3.7
+
+If you are using Python 3.6 and cannot upgrade to 3.7, install the
+``dataclasses`` backport:
+
+::
+
+ pip install dataclasses
+
+Installing the TOM Toolkit and Django
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+First, we recommend using a `virtual
+environment `__ for your
+project. This will keep your TOM python packages seperate from your
+system python packages.
+
+::
+
+ python3 -m venv tom_env/
+
+Now that we have created the virtual environment, we can activate it:
+
+::
+
+ source tom_env/bin/activate
+
+You should now see ``(tom_env)`` prepended to your terminal prompt.
+
+Now, install the TOM Toolkit:
+
+::
+
+ pip install tomtoolkit
+
+…and create a new project, just like in the tutorial:
+
+::
+
+ django-admin startproject mytom
+
+You should now have a fully functional standard Django installation
+inside the ``mytom`` folder, with the TOM dependencies installed as
+well.
+
+Getting started with the ``tom_setup`` script.
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+We need to add the ``tom_setup`` app to our project’s
+``INSTALLED_APPS``. Locate the ``settings.py`` file inside your project
+directory (usually in a subdirectory of the main folder,
+i.e. mytom/mytom/settings.py) and edit it so that it looks like this:
+
+.. code:: python
+
+ INSTALLED_APPS = [
+ 'django.contrib.admin',
+ 'django.contrib.auth',
+ 'django.contrib.contenttypes',
+ 'django.contrib.sessions',
+ 'django.contrib.messages',
+ 'django.contrib.staticfiles',
+ 'tom_setup',
+ ]
+
+Run the setup script
+~~~~~~~~~~~~~~~~~~~~
+
+The ``tom_setup`` app contains a script that will bootstrap a new TOM in
+your current project. Run it:
+
+::
+
+ ./manage.py tom_setup
+
+The install script will ask you a few questions and then install your
+TOM.
+
+Running the dev server
+~~~~~~~~~~~~~~~~~~~~~~
+
+Now that the toolkit is installed, you’re ready to try it out!
+
+First, run the necessary migrations:
+
+::
+
+ ./manage.py migrate
+
+Now, start the dev server:
+
+::
+
+ ./manage.py runserver
+
+Your new TOM should now be running on http://127.0.0.1:8000!
diff --git a/docs/introduction/index.rst b/docs/introduction/index.rst
index 79a1bdfac..f891498f1 100644
--- a/docs/introduction/index.rst
+++ b/docs/introduction/index.rst
@@ -10,6 +10,7 @@ Introduction
workflow
resources
faqs
+ troubleshooting
:doc:`Architecture ` - This document describes the architecture of the TOM Toolkit at a
high level. Read this first if you're interested in how the TOM Toolkit works.
@@ -21,4 +22,6 @@ high level. Read this first if you're interested in how the TOM Toolkit works.
:doc:`Programming Resources ` - Resources for learning the elements of programming used in the TOM Toolkit:
HTML, CSS, Python, and Django
-:doc:`Frequently Asked Questions ` - Look here for a potential quick answer to a common question.
\ No newline at end of file
+:doc:`Frequently Asked Questions ` - Look here for a potential quick answer to a common question.
+
+:doc:`Troubleshooting ` - Find solutions to common problems or information on how to debug an issue.
\ No newline at end of file
diff --git a/docs/introduction/resources.rst b/docs/introduction/resources.rst
index 54e0bb163..ee4eb64d5 100644
--- a/docs/introduction/resources.rst
+++ b/docs/introduction/resources.rst
@@ -31,7 +31,9 @@ Django
`Classy Class-Based Views `_ - Inheritance can be confusing. The CCBV reference page shows the class hierarchy of Django's built-in class-based views, along with the available attributes and methods, and the class they each come from.
-`Security in Django `_ - Take special note of
+`Databases `_ - Information on setting up different database backends in Django.
+
+`Security in Django `_ - Make sure your Django application is secure.
Python
diff --git a/docs/introduction/support.rst b/docs/introduction/support.rst
new file mode 100644
index 000000000..5a8b98d23
--- /dev/null
+++ b/docs/introduction/support.rst
@@ -0,0 +1,29 @@
+Getting Support
+===============
+
+This page will go over the process for reporting issues, requesting features, and getting
+support for the TOM Toolkit.
+
+Check the FAQ
+-------------
+
+Take a look at our :doc:`Frequently Asked Questions page ` for a potential quick answer to a common query.
+
+Reporting Issues
+----------------
+
+Issue reporting can be done via the `Github issues page `_
+of the tom_base project. Reporting an issue requires a Github account, but provides an easy way for
+developers to ask follow-up questions about an issue in order to resolve it.
+
+Please include as much detail as possible, as well as the steps taken that trigger the issue, and be sure to tag it with the "bug" tag!
+
+Requesting Features
+-------------------
+
+Like issues, feature and enhancement requests should be done via the same `Github issues page `_ of the tom_base project. This is also a great place to see what's being worked on and what's already been requested, which will allow you to voice support for a backlogged feature to be reprioritized.
+
+Support
+-------
+
+If you're looking for help with some aspect of your TOM, the `Github issues page `_ is once again the place to go. The "question" or "help wanted" tags should be very useful when looking for support, and the TOM Toolkit developers are more than happy to provide the help necessary to get your TOM running. You may also want to peruse the `Closed Issues `_, where someone may have already had (and solved!) your problem.
\ No newline at end of file
diff --git a/docs/introduction/tomarchitecture.md b/docs/introduction/tomarchitecture.rst
similarity index 53%
rename from docs/introduction/tomarchitecture.md
rename to docs/introduction/tomarchitecture.rst
index 883b77076..431797ca4 100644
--- a/docs/introduction/tomarchitecture.md
+++ b/docs/introduction/tomarchitecture.rst
@@ -1,9 +1,9 @@
TOM Software Architecture
--------------------------
+*************************
The goal of the TOM Toolkit is to make developing TOMs as easy as possible while
providing the flexibility needed to tailor each TOM to its specific science
-case. The motivation for the TOM Toolkit is discussed on the [about](https://tomtoolkit.github.io/about)
+case. The motivation for the TOM Toolkit is discussed on the :doc:`about `
page.
The TOM Toolkit (referred to as "the toolkit") provides a framework for
@@ -14,15 +14,15 @@ Web-based technologies allow developers to create rich
user interfaces, simplify distribution and choose from a huge variety of
programming languages and frameworks.
-[Python](https://python.org) has become the go-to language for many in
+`Python `_ has become the go-to language for many in
science. Fortunately, Python also enjoys widespread popularity in
web development communities. This provides a unique opportunity for "Pythonistas" to
develop scientific codebases which integrate seamlessly with web-based
technologies. One need look no further than the success of the
-[Jupyter](https://jupyter.org) project to see evidence of this.
+`Jupyter `_ project to see evidence of this.
There has been a lot of development surrounding Python and the web in the last
-two decades. One framework in particular, [Django](https://djangoproject.com)
+two decades. One framework in particular, `Django `_
has emerged as one of the more popular choices for web development. Django is
well known for its maturity, ease of use and modularity.
@@ -30,51 +30,38 @@ Instead of reinventing the wheel, it often makes sense to build on the proven
work of others. Thus, it was decided that the toolkit would build on top of the
Django framework. This provides several advantages:
-1. The toolkit does not need to re-implement generic functionality that Django
-already provides such as template rendering, routing, object relational mapping,
-or even higher-level functionality like user accounts and database migrations.
+#. The toolkit does not need to re-implement generic functionality that Django already provides such as template rendering, routing, object relational mapping, or even higher-level functionality like user accounts and database migrations.
-2. TOM developers get to take advantage of the massive amounts of existing
-knowledge that already exists for Django projects. In fact, much of the extra
-functionality that a TOM developer might want to implement need not be dependent
-on the the toolkit at all, but can instead be developed by referring to the
-[excellent documentation](https://docs.djangoproject.com/en/2.2/) Django
-provides.
+#. TOM developers get to take advantage of the massive amounts of existing knowledge that already exists for Django projects. In fact, much of the extra functionality that a TOM developer might want to implement need not be dependent on the the toolkit at all, but can instead be developed by referring to the `excellent documentation `_ Django provides.
-3. There are [thousands of Django packages](https://djangopackages.org) already
-written that can be used in any TOM project. If a TOM developer wants to be able
-to generate dynamic plots, or allow their users to login with Google, or even
-turn their TOM into a Slack bot, chances are there is already a package
-available that might suit their needs.
+#. There are `thousands of Django packages `_ already written that can be used in any TOM project. If a TOM developer wants to be able to generate dynamic plots, or allow their users to login with Google, or even turn their TOM into a Slack bot, chances are there is already a package available that might suit their needs.
We **highly recommend** that developers interested in utilizing the TOM Toolkit
familiarize themselves with the basics of Django, especially if they want to
-customize the toolkit in any significant fashion. The majority of the [guides
-found in the TOM toolkit documentation](/index) are simply Django concepts
-rewritten in a TOM context.
+customize the toolkit in any significant fashion. The majority of the :doc:`guides found in the TOM toolkit documentation ` are simply Django concepts rewritten in a TOM context.
-### Extending and Customizing the TOM Toolkit
+Extending and Customizing the TOM Toolkit
+=========================================
-As mentioned before, Django is well known for its extendibility and modularity.
+As mentioned before, Django is well known for its extensibility and modularity.
The toolkit takes advantage of these strengths heavily. In many ways, the TOM
Toolkit is a framework within a framework.
-After a TOM developer follows the [getting started guide](getting_started)
+After a TOM developer follows the :doc:`getting started guide `
they are left with a functioning but generic TOM. It is then up to the developer
to implement the specific features that their science case requires. The toolkit
tries to facilitate this as efficiently as possible and provides
-[documentation](/index) in areas of customization from [changing the HTML layout
-of a page](/customization/customize_templates) to [altering how observations are
-submitted](/customization/customize_observations) and even [creating a new alert
-broker](/customization/create_broker).
+:doc:`documentation ` in areas of customization from :doc:`changing the HTML layout of a page `
+to :doc:`altering how observations are submitted ` and even
+:doc:`creating a new alert broker `.
Django, and by extension the toolkit, rely heavily on object oriented
programming, especially inheritance. Most customization in the TOM toolkit comes
from subclassing classes that provide generic functionality and overriding or
extending methods. An experienced Django developer would feel right at home. For example, the
-[ObservationRecordDetailView](https://github.com/TOMToolkit/tom_base/blob/master/tom_observations/views.py#L143)
-in the `tom_observations` module of the toolkit inherits from Django's
-[DetailView](https://docs.djangoproject.com/en/2.2/ref/class-based-views/generic-display/#detailview).
+`ObservationRecordDetailView `_
+in the ``tom_observations`` module of the toolkit inherits from Django's
+`DetailView `_.
This means TOM developers are able to take full advantage of the power of Django
while still benefiting from the basic functionality that the toolkit provides.
@@ -82,36 +69,38 @@ This is why we recommend TOM developers familiarize themselves with Django; most
TOM Toolkit features are actually extended Django features.
-#### Plugin Architecture
+Plugin Architecture
+===================
+
Some areas of the TOM implement a plugin based architecture to support multiple
implementations of a similar functionality. An example would be the
-`tom_observations` module in which every supported observatory is implemented
-as its own plugin. The `tom_catalogs` and `tom_alerts` work in the same way: the
+`tom_observations`` module in which every supported observatory is implemented
+as its own plugin. The ``tom_catalogs`` and ``tom_alerts`` work in the same way: the
module defines the interface and generic functionality and each implementation
fills in its own logic.
This structure makes it easy for developers to write their own plugins which can
then be shared and installed by others or even contributed to the main codebase.
-The [gemini.py
-module](https://github.com/TOMToolkit/tom_base/blob/master/tom_observations/facilities/gemini.py)
+The `gemini.py module `_
is an observation module plugin contributed by Bryan Miller to enable the
triggering of observation requests on the Gemini telescope via the TOM Toolkit.
Thanks Bryan!
-#### Template Engine
-The toolkit is able to take advantage of Django's excellent [template
-engine](https://docs.djangoproject.com/en/2.2/topics/templates/). Part of the
+Template Engine
+===============
+
+The toolkit is able to take advantage of Django's excellent `template engine `_. Part of the
engine's power comes form the ability of templates to extend and override
each other. This means a TOM developer can easily change the layout and style of
any page without modifying the underlying framework's code directly. Entire pages
may be replaced, or only "blocks" within a template.
-Compare these screenshots of the [standard target detail
-page](../../../_static/architecture/standardlayout.png) and the [Global Supernova
-Project's target detail page](../../../_static/architecture/snex2layout.png), the
+Compare these screenshots of the `standard target detail page <../../../_static/architecture/snex2layout.png>`_ and the
+`Global Supernova Project's target detail page <../../../_static/architecture/snex2layout.png>`_, the
latter taking heavy advantage of template inheritance.
-### Data Storage, Deployment and Tooling
+Data Storage, Deployment and Tooling
+====================================
The toolkit is implemented as a web application backed by a relational database,
uses (mostly) server side rendering, and is deployed using wsgi.
@@ -123,30 +112,28 @@ ones. By default SQLite is deployed because of its ease of use.
For non-database storage (data products, fits files, etc) the toolkit can be
configured to use a variety of cloud-based storage services via
-[django-storages](https://django-storages.readthedocs.io). The documentation
-provides a guide for [storing data on Amazon S3](/deployment/amazons3). By default,
+`django-storages `_. The documentation
+provides a guide for :doc:`storing data on Amazon S3 `. By default,
data is stored on disk.
Similarly, deployment works with a variety of servers, including uWsgi and
-Gunicorn. The documentation provides a guide to [deploying to
-Heroku](/deployment/deployment_heroku) for those who want to get up and running
+Gunicorn. The documentation provides a guide to :doc:`deploying to Heroku ` for those who want to get up and running
quickly. Another option is to use Docker: the demo instance of the toolkit is
deployed to a Kubernetes cluster and the
-[Dockerfile](https://github.com/TOMToolkit/tom_demo/blob/master/Dockerfile) is
+`Dockerfile `_ is
available on Github.
-On the frontend, the toolkit utilizes the very popular [Bootstrap4 css
-framework](https://getbootstrap.com) for its layout and general look, making it
+On the frontend, the toolkit utilizes the very popular `Bootstrap4 css framework `_ for its layout and general look, making it
easy to pickup for anyone with experience with CSS. Javascript is introduced
sparingly (astronomers love Python!) but is used in various situations to
enhance the user experience and enable functionality such as interactive
plotting and sky maps.
-### Django Reusable Apps
+Django Reusable Apps
+====================
As previously mentioned, one of the reasons for Django's popularity is its
-modularity. Django has the concept of [reusable
-apps](https://docs.djangoproject.com/en/2.2/intro/reusable-apps/) which are just
+modularity. Django has the concept of `reusable apps `_ which are just
python packages that are specifically meant to be used inside a Django project.
The majority of the the toolkit's functionality is implemented in a series of
Django apps. While most of the apps are required, some may be omitted entirely
@@ -154,38 +141,39 @@ from a TOM if the functionality is not desired.
The following describes each app that ships with the toolkit and its purpose.
-#### TOM Targets
+TOM Targets
+-----------
-The
-[tom_targets](https://github.com/TOMToolkit/tom_base/tree/master/tom_targets)
+The `tom_targets `_
app is central to the entire TOM Toolkit project. It provides the database
definitions for the storage and retrieval of targets and target lists. It also
provides the views (pages) for viewing, creating, modifying and visualizing
these targets in several ways including the visibility and target distribution
plots.
-Nearly every app depends on the `tom_targets` module in some way.
+Nearly every app depends on the ``tom_targets`` module in some way.
-#### TOM Observations
+TOM Observations
+----------------
-The
-[tom_observations](https://github.com/TOMToolkit/tom_base/tree/master/tom_observations)
+The `tom_observations `_
app handles all the logic for submitting and querying observations of targets at
observatories. It defines the database models for observation requests and
provides some views for working with them.
-[facility.py](https://github.com/TOMToolkit/tom_base/blob/master/tom_observations/facility.py)
+`facility.py `_
defines an interface that external facilities (observatories) can implement in
order to integrate with the toolkit:
-[gemini.py](https://github.com/TOMToolkit/tom_base/blob/master/tom_observations/facilities/gemini.py)
+`gemini.py `_
and
-[lco.py](https://github.com/TOMToolkit/tom_base/blob/master/tom_observations/facilities/lco.py)
+`lco.py `_
are two examples, and we expect more in the future.
-#### TOM Data Products
+TOM Data Products
+-----------------
-Straddling both the `tom_targets` and `tom_observations` packages is
-[tom_dataproducts](https://github.com/TOMToolkit/tom_base/tree/master/tom_dataproducts).
+Straddling both the ``tom_targets`` and ``tom_observations`` packages is
+`tom_dataproducts `_.
This package contains the logic required for storing data related to targets and
observations within the toolkit. Some data products are fetched from on-line
archives (handled by an observatory's observation module) but data can also be
@@ -196,107 +184,118 @@ or in the cloud) as well as displaying certain kinds of data. It also provides
code hooks where TOM developers can run their own functions on the data in case
specialized data processing, analytics or pipelining is required.
-#### TOM Alerts
+TOM Alerts
+----------
-The [tom_alerts](https://github.com/TOMToolkit/tom_base/tree/master/tom_alerts)
+The `tom_alerts `_
app contains modules related to the functionality of ingesting targets from
various external services. These services, usually called brokers, provide
rapidly changing target lists that are of interest to time domain astronomers.
The
-[alerts.py](https://github.com/TOMToolkit/tom_base/blob/master/tom_alerts/alerts.py)
+`alerts.py `_
module provides a generic interface that other modules can implement, giving
them the ability to integrate these brokers with the toolkit. Currently, there are
-modules available for [Lasair](https://lasair.roe.ac.uk),
-[MARS](https://mars.lco.global) and
-[SCOUT](https://cneos.jpl.nasa.gov/scout/intro.html) with more planned for the
-future.
+modules available for `Lasair `_,
+`MARS `_, `SCOUT `_, and others,
+with more planned for the future.
-#### TOM Catalogs
+TOM Catalogs
+------------
The
-[tom_catalogs](https://github.com/TOMToolkit/tom_base/tree/master/tom_catalogs)
+`tom_catalogs `_
app contains functionality related to querying astronomical catalogs. These
"harvester" modules enable the querying and translation of targets found in
databases such as Simbad and JPL Horizons directly into targets within the
toolkit. The
-[harvester.py](https://github.com/TOMToolkit/tom_base/blob/master/tom_catalogs/harvester.py)
+`harvester.py `_
module provides the basic interface, and there are several modules already
written for Simbad, NED, the MPC, JPL Horizons and the Transient Name Server.
-#### TOM Setup and TOM Common
+TOM Setup and TOM Common
+------------------------
-The [tom_setup](https://github.com/TOMToolkit/tom_base/tree/master/tom_setup)
+The `tom_setup `_
package is special in that its sole purpose is to help TOM developers bootstrap
-new TOMs. See the [getting started](getting_started) guide for an example.
-The [tom_common](https://github.com/TOMToolkit/tom_base/tree/master/tom_common)
+new TOMs. See the :doc:`getting started ` guide for an example.
+The `tom_common `_
package contains logic and data that doesn't fit anywhere else.
-### Database Layout
+Database Layout
+---------------
+
The following diagram is an Entity-relationship Diagram (ERD). It is meant to
display the relationship between tables in a database. In this case, it may help
illustrate how the data from each of the toolkit's packages relate to each other.
It is not exhaustive; many tables and rows have been omitted for brevity.
-[](../_images/erd.png)
+.. image:: /_static/architecture/erd.png
+ :alt: DB Layout
+
+Models
+======
-### Models
Django models are the classes that map to the database tables in your Django application. The
TOM Toolkit models and the rationale behind them do are largely intuitive, but may require some
explanation.
-#### Target
-The `Target` model is relatively self-evident--it stores the data that describes the
+Target
+------
+
+The ``Target`` model is relatively self-evident--it stores the data that describes the
targets in your TOM. By default, that includes things like name, type, coordinates, and
ephemerides.
-#### TargetName
-The `TargetName` model stores extra names for a target, aka aliases. The corresponding target
+TargetName
+----------
+
+The ``TargetName`` model stores extra names for a target, aka aliases. The corresponding target
is stored as a foreign key.
-#### ObservationRecord
-The `ObservationRecord` model describes an individual observation request for a single target.
+ObservationRecord
+-----------------
+
+The ``ObservationRecord`` model describes an individual observation request for a single target.
It stores the target as a foreign key, and can optionally store facility information and the
parameters submitted for the observation.
-#### DataProduct
-The `DataProduct` model can refer to a number of different things, but generally refers to a
-single file that is associated with a `Target` and optionally an `ObservationRecord`. A
-`DataProduct` has one of a number of tags, which at present include the following:
+DataProduct
+-----------
+
+The ``DataProduct`` model can refer to a number of different things, but generally refers to a
+single file that is associated with a ``Target`` and optionally an ``ObservationRecord``. A
+`DataProduct`` has one of a number of tags, which at present include the following:
- Photometry, a file containing photometric data
- FITS, any FITS file not falling into the other categories
- Spectroscopy, a file containing spectroscopic data
- Image, a file containing image data, such as a JPEG or PNG
-A `DataProduct` type is file format-agnostic and refers to the data contained in the file,
+A ``DataProduct`` type is file format-agnostic and refers to the data contained in the file,
rather than the format itself. The type is necessary for making decisions on which operations
can be executed using the data in a file.
-#### ReducedDatum
-A `ReducedDatum` is a single point of data associated with a `Target` and optionally a
-`DataProduct`. The single data point is typically a single point of photometry or an individual
-spectrum. The `ReducedDatum` model has the following fields, in addition to its aforementioned
+ReducedDatum
+------------
+
+A ``ReducedDatum`` is a single point of data associated with a ``Target`` and optionally a
+``DataProduct``. The single data point is typically a single point of photometry or an individual
+spectrum. The ``ReducedDatum`` model has the following fields, in addition to its aforementioned
foreign key relationships:
-- `data_type` is maintained on both the `ReducedDatum` and `DataProduct` for the
-case when data is brought in from another source, such as a broker
-- The `source_name` optionally refers to the original source of the data. The
-intent of this field was to track data ingested from brokers, but could potentially be used for
-other purposes.
-- `source_location` optionally gives a hard location to the source--for a
-broker, it would be a link to the original alert.
-- The `timestamp` time at which the datum was produced.
-- `value` is a `TextField` that can take any series of data. As implemented, photometry
-is stored as JSON with keys for magnitude and error, but the `TextField` provides flexibility for
-additional photometry values on the datum. Spectroscopy is also stored as JSON, with keys for
-`magnitude` and `flux`.
-
-### Feedback and bug reporting
+- ``data_type`` is maintained on both the ``ReducedDatum`` and ``DataProduct`` for the case when data is brought in from another source, such as a broker
+- The ``source_name`` optionally refers to the original source of the data. The intent of this field was to track data ingested from brokers, but could potentially be used for other purposes.
+- ``source_location`` optionally gives a hard location to the source--for a broker, it would be a link to the original alert.
+- The ``timestamp`` time at which the datum was produced.
+- ``value`` is a ``TextField`` that can take any series of data. As implemented, photometry is stored as JSON with keys for magnitude and error, but the ``TextField`` provides flexibility for additional photometry values on the datum. Spectroscopy is also stored as JSON, with keys for ``magnitude`` and ``flux``.
+
+Feedback and bug reporting
+==========================
We hope the TOM Toolkit is helpful to you and your project. If you have any
concerns about implementation details, or questions about your own needs, please
-don't hesitate to [reach out](mailto:ariba@lco.global). Issues and pull requests
-are also welcome on the project's [GitHub page](https://github.com/TOMToolkit/).
+don't hesitate to `reach out `_. Issues and pull requests
+are also welcome on the project's `GitHub page `_.
diff --git a/docs/introduction/troubleshooting.rst b/docs/introduction/troubleshooting.rst
new file mode 100644
index 000000000..b08af9dc9
--- /dev/null
+++ b/docs/introduction/troubleshooting.rst
@@ -0,0 +1,52 @@
+Troubleshooting your TOM
+========================
+
+When first installing or later updating your TOM, you may run into a few
+common issues. Fortunately, you can stand on our shoulders and hopefully
+find a solution here!
+
+Check that you’ve migrated
+--------------------------
+
+Oftentimes, updating the TOM Toolkit requires running migrations.
+Usually, a directive to do so will be included in the release notes, or
+Django will remind you that ``You have unapplied migrations``. If you
+don’t happen to see those, you may also see a
+`` does not exist`` when you load a page, or an error
+about an ``applabel``. Those are generally indicators that you need to
+run a database migration.
+
+You can confirm that you are missing a migration by running:
+
+::
+
+ ./manage.py showmigrations --list
+
+Migrations that have been applied will have a ``[X]`` next to them, so
+make sure they all have one. If any are missing:
+
+::
+
+ ./manage.py migrate
+
+Make sure you’re in a virtual environment
+-----------------------------------------
+
+Everyone forgets to activate their virtualenv from time to time. If you
+get a missing package or some such, ensure that you’ve activated your
+virtualenv:
+
+::
+
+ source env/bin/activate
+
+You may need to adapt the above for your particular shell. Also be sure
+that the virtualenv was created with a compatible version of Python, and
+that you installed your dependencies into that virtualenv.
+
+Check your shell
+----------------
+
+It’s a small development team, and we all use bash. We’ve seen some
+issues with people running zsh, fish, and even csh. You may need to
+adapt the commands given in the setup guide.
diff --git a/docs/introduction/workflow.md b/docs/introduction/workflow.md
deleted file mode 100644
index 59ed01b2b..000000000
--- a/docs/introduction/workflow.md
+++ /dev/null
@@ -1,86 +0,0 @@
-TOM Workflow
----
-
-### Targets
-
-Targets are the central entity of the TOM Toolkit. Most functionality in the
-toolkit requires a target as they are the object of study. A target represents an
-astronomical object (star, galaxy, asteroid, etc) and is usually represented using
-coordinates on the sky along with other meta data.
-
-#### Creating Targets
-
-The TOM Toolkit provides a variety of methods for importing astronomical targets
-into the TOM:
-
-
-
-
-* The Alert Module provides the functionality to create targets from alert brokers
-such as [MARS](https://mars.lco.global) and [ANTARES](https://antares.noao.edu/).
-These brokers generally provide alerts from transient phenomena as soon as they
-happen, and a scientist who is interested in studying these phenomena can import
-these alerts as targets into their TOM to study in real time.
-
-* Online catalogs such as SIMBAD and the JPL Horizons contain information on
- millions of existing astronomical objects. If a scientist wishes to study one of
- these existing objects, they can query these catalogs directly from the TOM and
- use the returned data to create TOM Targets.
-
-* Manual entry/bulk upload allows a scientist to create targets that aren't known
- by any of the existing catalogs or use more precise information that they know
- of.
-
-
-### Observations
-
-After creating targets, the scientist needs to collect data on these targets. The
-TOM Observing module provides an interface to several observatories for which
-observations can be requested.
-
-#### Requesting Observations
-
-Using the TOM Observation module, scientists can request observations of their
-targets to one or many different observatories. Since the observing module has
-access to targets stored in the TOM database it can automatically fill in many of
-the observing parameters required by observing facilities, greatly reducing the
-workload of the scientist. The observing module also provides a common interface,
-removing the need of the scientist to navigate many different online systems to
-request observations.
-
-Observations can also be requested in a completely automated manner, which is
-particularly useful for rapid response time domain follow-up programs.
-
-
-
-
-#### Observation Status
-
-Once an observation for a target is created it's status is kept up to date within
-the TOM. When the status of an observation request at an observatory changes
-(failed, completed, postponed, etc) the scientist may be notified by the TOM.
-
-### Data
-
-The ultimate goal of the TOM toolkit is to collect and organize data. The TOM data
-module provides several methods for obtaining data, the most obvious being from
-completed observations. Scientists can also upload any data they'd like to
-associate with their targets as well.
-
-#### Data Processing
-
-The TOM toolkit provides a framework to write custom code to
-interact with the data the TOM obtains (among other things). These are called
-"hooks" and they can be used by scientists to write custom image pipelines, data
-quality checks, or to hook into entirely different systems. For example: if a
-scientist has existing code that checks images of microlensed stars for
-exoplanets, they may hook the code into the TOM toolkit directly to run whenever
-new data is acquired.
-
-#### Downloading Data
-
-Data is stored in the TOM toolkit by default, but many scientists may want to
-download the data somewhere else to do offline processing. Scientists can easily
-download data to their local machines, and the data module by default stores all
-it's data on a local file system. However, it can be customized to store data on
-cloud services, like Amazon S3, when desired.
diff --git a/docs/introduction/workflow.rst b/docs/introduction/workflow.rst
new file mode 100644
index 000000000..ba40683ab
--- /dev/null
+++ b/docs/introduction/workflow.rst
@@ -0,0 +1,98 @@
+TOM Workflow
+------------
+
+Targets
+~~~~~~~
+
+Targets are the central entity of the TOM Toolkit. Most functionality in
+the toolkit requires a target as they are the object of study. A target
+represents an astronomical object (star, galaxy, asteroid, etc) and is
+usually represented using coordinates on the sky along with other meta
+data.
+
+Creating Targets
+^^^^^^^^^^^^^^^^
+
+The TOM Toolkit provides a variety of methods for importing astronomical
+targets into the TOM:
+
+.. image:: /_static/target_sources.png
+
+- The Alert Module provides the functionality to create targets from
+ alert brokers such as ``MARS ``\ \_\_ and
+ ``ANTARES ``\ \__. These brokers generally
+ provide alerts from transient phenomena as soon as they happen, and a
+ scientist who is interested in studying these phenomena can import
+ these alerts as targets into their TOM to study in real time.
+
+- Online catalogs such as SIMBAD and the JPL Horizons contain
+ information on millions of existing astronomical objects. If a
+ scientist wishes to study one of these existing objects, they can
+ query these catalogs directly from the TOM and use the returned data
+ to create TOM Targets.
+
+- Manual entry/bulk upload allows a scientist to create targets that
+ aren’t known by any of the existing catalogs or use more precise
+ information that they know of.
+
+Observations
+~~~~~~~~~~~~
+
+After creating targets, the scientist needs to collect data on these
+targets. The TOM Observing module provides an interface to several
+observatories for which observations can be requested.
+
+Requesting Observations
+^^^^^^^^^^^^^^^^^^^^^^^
+
+Using the TOM Observation module, scientists can request observations of
+their targets to one or many different observatories. Since the
+observing module has access to targets stored in the TOM database it can
+automatically fill in many of the observing parameters required by
+observing facilities, greatly reducing the workload of the scientist.
+The observing module also provides a common interface, removing the need
+of the scientist to navigate many different online systems to request
+observations.
+
+Observations can also be requested in a completely automated manner,
+which is particularly useful for rapid response time domain follow-up
+programs.
+
+.. image:: /_static/common_interface.png
+
+Observation Status
+^^^^^^^^^^^^^^^^^^
+
+Once an observation for a target is created it’s status is kept up to
+date within the TOM. When the status of an observation request at an
+observatory changes (failed, completed, postponed, etc) the scientist
+may be notified by the TOM.
+
+Data
+~~~~
+
+The ultimate goal of the TOM toolkit is to collect and organize data.
+The TOM data module provides several methods for obtaining data, the
+most obvious being from completed observations. Scientists can also
+upload any data they’d like to associate with their targets as well.
+
+Data Processing
+^^^^^^^^^^^^^^^
+
+The TOM toolkit provides a framework to write custom code to interact
+with the data the TOM obtains (among other things). These are called
+“hooks” and they can be used by scientists to write custom image
+pipelines, data quality checks, or to hook into entirely different
+systems. For example: if a scientist has existing code that checks
+images of microlensed stars for exoplanets, they may hook the code into
+the TOM toolkit directly to run whenever new data is acquired.
+
+Downloading Data
+^^^^^^^^^^^^^^^^
+
+Data is stored in the TOM toolkit by default, but many scientists may
+want to download the data somewhere else to do offline processing.
+Scientists can easily download data to their local machines, and the
+data module by default stores all it’s data on a local file system.
+However, it can be customized to store data on cloud services, like
+Amazon S3, when desired.
diff --git a/docs/managing_data/customizing_data_processing.rst b/docs/managing_data/customizing_data_processing.rst
new file mode 100644
index 000000000..62b295535
--- /dev/null
+++ b/docs/managing_data/customizing_data_processing.rst
@@ -0,0 +1,177 @@
+Customizing Data Processing
+---------------------------
+
+One of the many goals of the TOM Toolkit is to enable the simplification
+of the flow of your data from observations. To that end, there’s some
+built-in functionality that can be overridden to allow your TOM to work
+for your use case.
+
+To begin, here’s a brief look at part of the structure of the
+tom_dataproducts app in the TOM Toolkit:
+
+::
+
+ tom_dataproducts
+ ├──hooks.py
+ ├──models.py
+ └──processors
+ ├──data_serializers.py
+ ├──photometry_processor.py
+ └──spectroscopy_processor.py
+
+Let’s start with a quick overview of ``models.py``. The file contains
+the Django models for the dataproducts app–in our case, ``DataProduct``
+and ``ReducedDatum``. The ``DataProduct`` contains information about
+uploaded or saved ``DataProducts``, such as the file name, file path,
+and what kind of file it is. The ``ReducedDatum`` contains individual
+science data points that are taken from the ``DataProduct`` files.
+Examples of ``ReducedDatum`` points would be individual photometry
+points or individual spectra.
+
+Each ``DataProduct`` also has a ``data_product_type``. The
+``data_product_type`` is simply a description of what the file is, more
+or less, and is customizable. The list of supported
+``data_product_type``\ s is maintained in ``settings.py``:
+
+.. code:: python
+
+ # Define the valid data product types for your TOM. Be careful when removing items, as previously valid types will no
+ # longer be valid, and may cause issues unless the offending records are modified.
+ DATA_PRODUCT_TYPES = {
+ 'photometry': ('photometry', 'Photometry'),
+ 'fits_file': ('fits_file', 'FITS File'),
+ 'spectroscopy': ('spectroscopy', 'Spectroscopy'),
+ 'image_file': ('image_file', 'Image File')
+ }
+
+In order to add new data product types, simply add a new key/value pair,
+with the value being a 2-tuple. The first tuple item is the database
+value, and the second is the display value.
+
+All data products are automatically “processed” on upload, as well. Of
+course, that can mean different things to different TOMs! The TOM has
+two built-in data processors, both of which simply ingest the data into
+the database, and those are also specified in ``settings.py``:
+
+.. code:: python
+
+ DATA_PROCESSORS = {
+ 'photometry': 'tom_dataproducts.processors.photometry_processor.PhotometryProcessor',
+ 'spectroscopy': 'tom_dataproducts.processors.spectroscopy_processor.SpectroscopyProcessor',
+ }
+
+When a user either uploads a ``DataProduct`` to their TOM, the TOM runs
+``process_data()`` from the corresponding ``DataProcessor`` subclass
+specified in ``DATA_PROCESSORS`` seen above. To illustrate, this is the
+base ``DataProcessor`` class:
+
+.. code:: python
+
+ import mimetypes
+
+ ...
+
+ class DataProcessor():
+
+ FITS_MIMETYPES = ['image/fits', 'application/fits']
+ PLAINTEXT_MIMETYPES = ['text/plain', 'text/csv']
+
+ mimetypes.add_type('image/fits', '.fits')
+ mimetypes.add_type('image/fits', '.fz')
+ mimetypes.add_type('application/fits', '.fits')
+ mimetypes.add_type('application/fits', '.fz')
+
+ def process_data(self, data_product):
+ pass
+
+Now let’s look at the built-in data processors. First, let’s check out
+the ``PhotometryProcessor``, which inherits from ``DataProcessor``:
+
+.. code:: python
+
+ class PhotometryProcessor(DataProcessor):
+
+ def process_data(self, data_product):
+ mimetype = mimetypes.guess_type(data_product.data.path)[0]
+ if mimetype in self.PLAINTEXT_MIMETYPES:
+ photometry = self._process_photometry_from_plaintext(data_product)
+ return [(datum.pop('timestamp'), json.dumps(datum)) for datum in photometry]
+ else:
+ raise InvalidFileFormatException('Unsupported file type')
+
+This class has an implementation of ``process_data()`` from the
+superclass ``DataProcessor``. The implementation calls an internal
+method ``_process_photometry_from_plaintext()``, which return a ``list``
+of ``dict``\ s. Each dict contains the values for the timestamp,
+magnitude, filter, and error for that photometry point. The list is then
+transformed into a list of 2-tuples, with the first value being the
+photometry timestamp, and the second being the JSON-ified remaining
+values.
+
+Next, let’s look at the ``SpectroscopyProcessor``:
+
+.. code:: python
+
+ class SpectroscopyProcessor(DataProcessor):
+
+ DEFAULT_WAVELENGTH_UNITS = units.angstrom
+ DEFAULT_FLUX_CONSTANT = units.erg / units.cm ** 2 / units.second / units.angstrom
+
+ def process_data(self, data_product):
+
+ mimetype = mimetypes.guess_type(data_product.data.path)[0]
+ if mimetype in self.FITS_MIMETYPES:
+ spectrum, obs_date = self._process_spectrum_from_fits(data_product)
+ elif mimetype in self.PLAINTEXT_MIMETYPES:
+ spectrum, obs_date = self._process_spectrum_from_plaintext(data_product)
+ else:
+ raise InvalidFileFormatException('Unsupported file type')
+
+ serialized_spectrum = SpectrumSerializer().serialize(spectrum)
+
+ return [(obs_date, serialized_spectrum)]
+
+Just like the ``PhotometryProcessor``, this class inherits from
+``DataProcessor`` and implements ``process_data()``. This is a
+requirement for a custom DataProcessor! This ``process_data()`` method
+handles two file types, unlike the previous example, each of which calls
+an internal method that returns a ``Spectrum1D`` object. Again, like the
+``PhotometryProcessor``, a list of 2-tuples is created, with the first
+value being the timestamp, and the second being the JSON spectrum.
+
+You may be wondering why these two methods return lists of 2-tuples,
+especially when the ``SpectroscopyProcessor`` only returns a list of
+length one. The rationale is to ensure that you, the TOM user, shouldn’t
+have to worry about the database insertion, so the internal logic
+handles that aspect, and it can do so whether you return one data point
+or many data points.
+
+For a custom ``DataProcessor``, there are just a few required steps. The
+first is to create a class that implements ``DataProcessor``, like so:
+
+.. code:: python
+
+ from tom_dataproducts.data_processor import DataProcessor
+
+
+ class MyDataProcessor(DataProcessor):
+
+ def process_data(self, data_product):
+ # custom data processing here
+
+ return [(timestamp1, json1), (timestamp2, json2), ..., (timestampN, dictN)]
+
+Let’s say that this file lives at ``mytom/my_data_processor.py``. Now
+the processor needs to be added to ``DATA_PROCESSORS``, and it can
+either process a new data product type, or replace an existing one.
+Let’s replace spectroscopy:
+
+.. code:: python
+
+ DATA_PROCESSORS = {
+ 'photometry': 'tom_dataproducts.processors.photometry_processor.PhotometryProcessor',
+ 'spectroscopy': 'mytom.my_data_processor.MyDataProcessor',
+ }
+
+And that’s it! Now your TOM will run the data processing specific to
+your case instead of the default one.
diff --git a/docs/managing_data/index.rst b/docs/managing_data/index.rst
new file mode 100644
index 000000000..86a7108d4
--- /dev/null
+++ b/docs/managing_data/index.rst
@@ -0,0 +1,18 @@
+Managing Data
+=============
+
+.. toctree::
+ :maxdepth: 2
+ :hidden:
+
+
+ ../api/tom_dataproducts/views
+ plotting_data
+ customizing_data_processing
+
+
+:doc:`Creating Plots from TOM Data ` - Learn how to create plots using plot.ly and your TOM
+data to display anywhere in your TOM.
+
+:doc:`Adding Custom Data Processing ` - Learn how you can process data into your
+TOM from uploaded data products.
diff --git a/docs/managing_data/plotting_data.rst b/docs/managing_data/plotting_data.rst
new file mode 100644
index 000000000..57ae08537
--- /dev/null
+++ b/docs/managing_data/plotting_data.rst
@@ -0,0 +1,206 @@
+Plotting Data
+-------------
+
+The TOM Toolkit provides a few basic plots, such as photometry,
+spectroscopy and target distribution. Sometimes it would be useful to
+visualize data in a different way.
+
+In this tutorial you will learn how to build and display a very simple
+plot in our TOM: number of reduced data per target. The end result will
+demonstrate how to create a `plot.ly `__ plot with data
+from our TOM. You will even package the code in it’s own app so we can
+share it with other TOM users that might find it useful.
+
+If you haven’t already read the documentation on `customizing
+templates `__ you should read it
+first. You’ll need to edit a template in order to view your new plot
+somewhere.
+
+First, start a new app in our project to house the new plot (and perhaps
+other additions!):
+
+::
+
+ ./manage.py startapp myplots
+
+This will create a new `Django
+app `__
+in your project named myplots:
+
+::
+
+ myplots
+ ├── admin.py
+ ├── apps.py
+ ├── __init__.py
+ ├── migrations
+ │ └── __init__.py
+ ├── models.py
+ ├── tests.py
+ └── views.py
+
+ 1 directory, 7 files
+
+Note you don’t necessarily have to start a new app. If you’ve already
+started an app that you’d like to reuse, that works too.
+
+Now install the new app into your project’s settings.py file:
+
+.. code:: python
+
+ INSTALLED_APPS = [
+ 'django.contrib.admin',
+ 'django.contrib.auth',
+ ...
+ 'myplots',
+ ]
+
+Now that the ``myplots`` app is installed, create the directories
+necessary to contain your new plot:
+
+::
+
+ mkdir -p myplots/templates/myplots
+ mkdir myplots/templatetags
+
+The templates directory will contain the html template you can include
+in other templates to display your plot. The templatetags directory will
+contain the python code to construct the plot.ly plot.
+
+Start by creating the
+`templatetags `__
+file:
+
+::
+
+ touch myplots/templatetags/myplots_tags.py
+
+Edit this file, starting with the necessary imports:
+
+.. code:: python
+
+ from plotly import offline
+ import plotly.graph_objs as go
+ from django import template
+
+ from tom_targets.models import Target
+
+The ``plotly`` imports are needed for building an offline plot. The
+django ``template`` import gives access to the template library, which
+will allow for registering the template tag. Finally, the TOM Toolkit
+``Target`` class will allow access to the ``Target`` model (for
+querying).
+
+Next, add the boiler plate code for a template tag:
+
+.. code:: python
+
+ register = template.Library()
+
+
+ @register.inclusion_tag('myplots/targets_reduceddata.html')
+ def targets_reduceddata(targets=Target.objects.all()):
+
+First we instantiate the ``register`` decorator. You don’t need to know
+much about this other that it allows us to register functions as
+templatetags. The function ``targets_reduceddata`` is decorated with the
+``register`` decorator, which takes as an argument the template to
+render. The function definition takes in a queryset of ``Target``\ s as
+a keyword argument, but if none are supplied, defaults to all
+``Target``\ s in the database.
+
+Next, add the function body:
+
+.. code:: python
+
+ # order targets by creation date
+ targets = targets.order_by('-created')
+ # x axis: target names. y axis: datum count
+ data = [go.Bar(
+ x=[target.name for target in targets],
+ y=[target.reduceddatum_set.count() for target in targets]
+ )]
+ # Create the plot
+ figure = offline.plot(go.Figure(data=data), output_type='div', show_link=False)
+ # Add plot to the template context
+ return {'figure': figure}
+
+As the comments describe, the function code iterates over each
+``Target`` in the ``targets`` queryset adding the target name and datum
+count as x/y values to the ``Bar`` data structure. Check out the
+`plot.ly bar chart documentation `__
+for more information about the options available to you. As an exercise,
+try changing the values in the y axis. Or you could use a different
+chart type.
+
+Finally, the code adds the plot.ly plot to the template rendering
+context. Next we will create this template where this context will be
+rendered.
+
+Create the file, making sure it matches the template name specified in
+the template tag definition beforehand:
+
+::
+
+ touch myplots/templates/myplots/targets_reduceddata.html
+
+This file contains the simple contents:
+
+::
+
+ {% raw %}
+ {{ figure|safe }}
+ {% endraw %}
+
+All this template does is output the ``figure`` variable, which is the
+html generated from plotly in the templatetag. We also tell django that
+the output is safe, so that it doesn’t escape the html. That’s it.
+
+**Note:** If you’re running the development server, restart it now.
+Django doesn’t automatically pick up new templatetags.
+
+Now that the templatetag and template are complete, we can use it in any
+template. You might have your own templates which you’d like to add the
+plot to, or perhaps you’ve customized one of the TOM supplied templates
+as per the `customizing
+templates `__ documentation. Either
+way, including the templatetag works the same way. At the top of the
+template (after any ‘extends’) load the new tag library:
+
+::
+
+ {% raw %}
+ {% load myplots_tags %}
+ {% endraw %}
+
+Now insert the templatetag somewhere in the template where you’d like it
+to appear:
+
+::
+
+ {% raw %}
+ {% targets_reduceddata %}
+ {% endraw %}
+
+If your parent template already has a queryset of targets available in
+the context (for example, a target list page) you can pass it in to be
+used in your plot:
+
+::
+
+ {% raw %}
+ {% targets_reduceddata targets %}
+ {% endraw %}
+
+Otherwise the plot will simply use all targets in your database. Either
+way, you should end up with something like this:
+
+|image0|
+
+That’s it! Plot.ly provides a wide range of plotting capabilities, you
+should reference `the documentation `__ for
+more information. It would also be helpful to read `Django’s
+ORM `__ to become
+familiarized with wide range of methods of querying data.
+
+.. |image0| image:: /_static/plotting_data_doc/plot.png
diff --git a/docs/observing/customize_observations.rst b/docs/observing/customize_observations.rst
new file mode 100644
index 000000000..93d060120
--- /dev/null
+++ b/docs/observing/customize_observations.rst
@@ -0,0 +1,323 @@
+Changing How Observations are Submitted
+---------------------------------------
+
+The LCO Observation module for the TOM Toolkit ships with a default HTML
+form that facilitates submitting basic observations to the LCO network.
+It may sometimes be desirable to customize the form to show or hide
+fields, add new parameters, or change the submission logic itself,
+depending on the needs of the project. In this tutorial we will
+customize our LCO module to submit multiple observations with different
+filters at the same time.
+
+This guide assumes you have followed the `getting
+started `__ guide and have a working TOM
+up and running.
+
+Create a new Observation Module
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Many methods of customizing the TOM Toolkit involve inheriting/extending
+existing functionality. This time will be no different: we’ll crate a
+new observation module that inherits the existing functionality from
+``tom_observations.facilities.LCOFacility``.
+
+First, create a python file somewhere in your project to house your new
+module. For example it could live next to your ``settings.py``, or if
+you’ve started a new app, it could live there. It doesn’t really matter,
+as long as it’s located somewhere in your project:
+
+::
+
+ touch mytom/mytom/lcomultifilter.py
+
+Now add some code to this file to create a new observation module:
+
+.. code:: python
+
+ # lcomultifilter.py
+ from tom_observations.facilities.lco import LCOFacility
+
+
+ class LCOMultiFilterFacility(LCOFacility):
+ name = 'LCOMultiFilter'
+
+So what does the above code do?
+
+1. Line 1 imports the LCOFacility that is already shipped with the TOM
+ Toolkit. We want this class because it contains functionality we will
+ re-use in our own implementation.
+2. Line 4 defines a new class named ``LCOMultiFilterFacility`` that
+ inherits from ``LCOFacility``.
+3. Line 5 sets the name attribute of this class to ``LCOMultiFilter``.
+
+What you have done is created a new observation module that is
+functionally identical to the existing LCO module, but has a different
+name: ``LCOMultiFilter``. A good start!
+
+Now we need to tell our TOM where to find our new module so we can use
+it to submit observations. Add (or edit) the following lines to your
+``settings.py``:
+
+.. code:: python
+
+ # settings.py
+ TOM_FACILITY_CLASSES = [
+ 'tom_observations.facilities.lco.LCOFacility',
+ 'tom_observations.facilities.gemini.GEMFacility',
+ 'mytom.lcomultifilter.LCOMultiFilterFacility',
+ ]
+
+This code lists all of the observation modules that should be available
+to our TOM.
+
+With that done, go to any target in your TOM and you should see your new
+module in the list:
+
+|image0|
+
+You could now use the new module now to make an observation, and it
+would work the same as the old LCO module.
+
+Note that if you see an error like: “There was a problem authenticating
+with LCO” then you need to `add your LCO api
+key `__ to your ``settings.py`` file.
+
+Adding additional fields
+~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now that you’ve created a new observation module that’s functionally the
+same as the old LCO module, how do we change it? One thing that might be
+useful is to add some extra fields to the form: two more choices of
+filters and exposure times. Back in the ``lcomultifilter.py`` file add a
+new import and create a new class that will become the new form:
+
+.. code:: python
+
+ # lcomultifilter.py
+ from tom_observations.facilities.lco import LCOFacility, LCOObservationForm, filter_choices
+ from django import forms
+
+
+ class LCOMultiFilterForm(LCOObservationForm):
+ filter2 = forms.ChoiceField(choices=filter_choices)
+ exposure_time2 = forms.FloatField(min_value=0.1)
+ filter3 = forms.ChoiceField(choices=filter_choices)
+ exposure_time3 = forms.FloatField(min_value=0.1)
+
+
+ class LCOMultiFilterFacility(LCOFacility):
+ name = 'LCOMultiFilter'
+ form = LCOMultiFilterForm
+
+There is now a new class, ``LCOMultiFilterForm`` which inherits from
+``LCOObservationForm``, the form for the default interface. Additionally
+there are definitions for 4 fields: ``fiter2``, ``exposure_time2``,
+``filter3``, and ``exposure_time3``.
+
+A ``form`` attribute has been added on the ``LCOMultiFilterFacility``
+class, this tells our observation module to use the new
+``LCOMultiFilterForm`` instead of the default LCO observation form.
+
+Modifying the form layout
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Now that the desired fields have been added to the
+``LCOMultiFilterForm``, the form’s layout needs to be modified in order
+to actually display them. In this example we’ll split the form into two
+rows: one row for the three filter choices and exposure times, and
+another row for everything else. Note that the default form already has
+fields for ``filter`` and ``exposure_time``, so we’ll overwrite the
+entire layout so that they appear next to the new fields we added.
+
+The ``LCOObservationForm`` has a method ``layout()`` that returns the
+desired layout using the `crispy forms
+Layout `__
+class. Familiarizing yourself with the basic functionality of crispy
+forms would be a good idea if you wish to deeply customize your
+observation module’s form.
+
+With our modified layout added, the ``lcomultifilter.py`` file now looks
+like this:
+
+.. code:: python
+
+ # lcomultifilter.py
+ from tom_observations.facilities.lco import LCOFacility, LCOObservationForm, filter_choices
+ from django import forms
+ from crispy_forms.layout import Div
+
+
+ class LCOMultiFilterForm(LCOObservationForm):
+ filter2 = forms.ChoiceField(choices=filter_choices)
+ exposure_time2 = forms.FloatField(min_value=0.1)
+ filter3 = forms.ChoiceField(choices=filter_choices)
+ exposure_time3 = forms.FloatField(min_value=0.1)
+
+ def layout(self):
+ return Div(
+ Div(
+ Div(
+ 'name', 'proposal', 'ipp_value', 'observation_type', 'start', 'end',
+ css_class='col'
+ ),
+ Div(
+ 'instrument_name', 'exposure_count', 'max_airmass',
+ css_class='col'
+ ),
+ css_class='form-row'
+ ),
+ Div(
+ Div(
+ 'filter', 'exposure_time',
+ css_class='col'
+ ),
+ Div(
+ 'filter2', 'exposure_time2',
+ css_class='col'
+ ),
+ Div(
+ 'filter3', 'exposure_time3',
+ css_class='col'
+ ),
+ css_class='form-row'
+ )
+ )
+
+
+ class LCOMultiFilterFacility(LCOFacility):
+ name = 'LCOMultiFilter'
+ form = LCOMultiFilterForm
+
+Take a look at the layout and compare it to the `existing lco
+layout `__.
+A second row has been added that includes all the filter choices. Note
+that the original ``filter`` and ``exposure_time`` have been moved from
+their original location to the new row.
+
+Now if you select “LCOMultiFilter” from the list of observation
+facilities on a target you should see your new form:
+
+|image1|
+
+Is the form still too ugly for you? Trying playing with the layout
+definition to suit your needs.
+
+Changing the form submission behavior
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you are not familiar with the `LCO submission
+API `__ now might be a good
+time to take a look. The LCO Observation module uses this API to submit
+observations using the data provided in the form, so we need to modify
+how this happens. More specifically, we’d like to add two additional
+``Configuration`` to our observation request, one for each of our
+additional filters and exposure times.
+
+Using the ``observation_payload()`` method, we can use ``super()`` to
+get the original LCO module’s observation request, then modify it to
+suit the needs of our ``LCOMultiFilter`` class:
+
+.. code:: python
+
+ #lcomultifilter.py
+ from tom_observations.facilities.lco import LCOFacility, LCOObservationForm, filter_choices
+ from django import forms
+ from crispy_forms.layout import Div
+ from copy import deepcopy
+
+ class LCOMultiFilterForm(LCOObservationForm):
+ filter2 = forms.ChoiceField(choices=filter_choices)
+ exposure_time2 = forms.FloatField(min_value=0.1)
+ filter3 = forms.ChoiceField(choices=filter_choices)
+ exposure_time3 = forms.FloatField(min_value=0.1)
+
+ def layout(self):
+ return Div(
+ Div(
+ Div(
+ 'name', 'proposal', 'ipp_value', 'observation_type', 'start', 'end',
+ css_class='col'
+ ),
+ Div(
+ 'instrument_type', 'exposure_count', 'max_airmass',
+ css_class='col'
+ ),
+ css_class='form-row'
+ ),
+ Div(
+ Div(
+ 'filter', 'exposure_time',
+ css_class='col'
+ ),
+ Div(
+ 'filter2', 'exposure_time2',
+ css_class='col'
+ ),
+ Div(
+ 'filter3', 'exposure_time3',
+ css_class='col'
+ ),
+ css_class='form-row'
+ )
+ )
+
+ def observation_payload(self):
+ payload = super().observation_payload()
+ configuration2 = deepcopy(payload['requests'][0]['configurations'][0])
+ configuration3 = deepcopy(payload['requests'][0]['configurations'][0])
+ configuration2['instrument_configs'][0]['optical_elements']['filter'] = self.cleaned_data['filter2']
+ configuration2['instrument_configs'][0]['exposure_time'] = self.cleaned_data['exposure_time2']
+ configuration3['instrument_configs'][0]['optical_elements']['filter'] = self.cleaned_data['filter3']
+ configuration3['instrument_configs'][0]['exposure_time'] = self.cleaned_data['exposure_time3']
+ payload['requests'][0]['configurations'].extend([configuration2, configuration3])
+ return payload
+
+
+ class LCOMultiFilterFacility(LCOFacility):
+ name = 'LCOMultiFilter'
+ form = LCOMultiFilterForm
+
+Let’s go over what we did in this new ``observation_payload()`` method:
+
+1. Line 1: We call ``super().observation_payload()`` to get the
+ observation request which the parent class (LCOFacility) would have
+ called.
+2. Line 2-3 We copy the Request’s Configuration into two new
+ Configurations: ``configuration2`` and ``configuration3``. These will
+ be the additional Configuration we send to LCO.
+3. Lines 5-8: We set the value of these new Configuration ``filter`` and
+ ``exposure_time`` to the values we collected from our custom form.
+4. lines 10-11: Finally, we extend the original Request’s Configuration
+ array to include the 2 new Configuration we built. Return it and
+ we’re done!
+
+If you submit an observation request with the ``LCOMultiFilter``
+observation module now you should see that it creates an observation
+request with LCO with three Configuration!
+
+Summary
+~~~~~~~
+
+Our original requirement was to be able to submit observations to LCO
+with some additional filters and exposure times. We accomplished this
+by:
+
+1. Creating a new observation module: a ``LCOMultiFilterFacility`` class
+ and a ``LCOMultiFilterForm``, both of which were child classes of the
+ original ``LCOFacility`` class (since we wanted to keep most of the
+ functionality intact) and then added this new class to our
+ ``TOM_FACILITY_CLASSES`` setting.
+
+2. We added a few fields to ``LCOMultiFilterForm`` and modified it’s
+ layout to include these new fields using ``layout()``.
+
+3. We implemented the ``LCOMultiFilterForm`` ``observation_payload()``
+ which used the parent’s class return value and then modified it to
+ suit our needs.
+
+This is a good example of Object Oriented Programming in Python. If you
+are curious about how this all works, we recommend reading up on OOP in
+general, as well as how objects in Python 3 work.
+
+.. |image0| image:: /_static/customize_observations/observebutton.png
+.. |image1| image:: /_static/customize_observations/newform.png
diff --git a/docs/observing/index.rst b/docs/observing/index.rst
new file mode 100644
index 000000000..ffa879be8
--- /dev/null
+++ b/docs/observing/index.rst
@@ -0,0 +1,29 @@
+Observing Facilities and Observations
+=====================================
+
+.. toctree::
+ :maxdepth: 2
+ :hidden:
+
+ customize_observations
+ ../common/scripts
+ strategies
+ observation_module
+ ../api/tom_observations/facilities
+ ../api/tom_observations/views
+
+
+:doc:`Changing Request Submission Behavior ` - Learn how to customize observation forms
+in order to add additional parameters to observation requests.
+
+`Programmatically Submitting Observations <../common/scripts.html#creating-observations-programmatically>`__
+
+:doc:`Cadence and Observing Strategies ` - Learn how to build cadence strategies that submit observations based on
+the result of prior observations, as well as how to leverage observing templates to submit observations with fewer clicks.
+
+:doc:`Building a TOM Observation Facility Module ` - Learn to build a module which will
+allow your TOM to submit observation requests to observatories.
+
+:doc:`Facility Modules <../api/tom_observations/facilities>` - Take a look at the supported facilities.
+
+:doc:`Observation Views <../api/tom_observations/views>` - Familiarize yourself with the available Observation Views.
diff --git a/docs/observing/observation_module.rst b/docs/observing/observation_module.rst
new file mode 100644
index 000000000..996680f7e
--- /dev/null
+++ b/docs/observing/observation_module.rst
@@ -0,0 +1,321 @@
+Writing an observation module to interface with observatories
+=============================================================
+
+This guide will walk you through how to create a custom observation
+facility module using some mocked up endpoints to simulate a real
+observatory interface. It will also provide information on creating a
+custom manual observation facility for tracking observations that were
+not created through an API.
+
+You can use this example as the foundation to build an observing
+facility module to connect to a real observatory or track observations
+on non-API supported facilities.
+
+Be sure you’ve followed the `Getting
+Started `__ guide before continuing onto
+this tutorial.
+
+What is a observing facility module?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A TOM Toolkit observing facility module is a python module which
+contains the code necessary to provide an interface to an observing
+facility in a TOM. Some examples of existing modules are the `Las
+Cumbres
+Observatory `__
+and the
+`Gemini `__
+modules. Both allow the submission of observation requests to their
+respective observatories through a TOM.
+
+Prerequisites
+~~~~~~~~~~~~~
+
+You should have a working TOM already. You can start where the `Getting
+Started `__ guide leaves off. You should
+also be familiar with the observing facility’s API that you would like
+to work with.
+
+Creating a custom robotic facility
+----------------------------------
+
+Defining the minimal implementation
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Within any existing module in your TOM you should create a new python
+module (file) named ``myfacility.py``. For example, if you have a fresh
+TOM installation you’ll have a directory structure that looks something
+like this:
+
+::
+
+ ├── data
+ ├── db.sqlite3
+ ├── manage.py
+ ├── mytom
+ │ ├── __init__.py
+ │ ├── settings.py
+ │ ├── urls.py
+ │ └── wsgi.py
+ ├── static
+ ├── templates
+ └── tmp
+
+We’ll place our ``myfacility.py`` file inside the ``mytom`` directory,
+next to ``settings.py``. For now, copy the following lines into
+``myfacility.py``:
+
+.. code:: python
+
+ from tom_observations.facility import BaseRoboticObservationFacility, BaseRoboticObservationForm
+
+
+ class MyObservationFacilityForm(BaseRoboticObservationForm):
+ pass
+
+
+ class MyObservationFacility(BaseRoboticObservationFacility):
+ name = 'MyFacility'
+ observation_types = [('OBSERVATION', 'Custom Observation')]
+
+We’ll go over what these lines mean soon. First, we’ll add a setting to
+our project’s ``settings.py`` to tell the TOM Toolkit to use our new
+class:
+
+.. code:: python
+
+ TOM_FACILITY_CLASSES = [
+ 'tom_observations.facilities.lco.LCOFacility',
+ 'tom_observations.facilities.gemini.GEMFacility',
+ 'mytom.myfacility.MyObservationFacility'
+ ]
+
+Now go ahead and view a target in your TOM, you should see something
+like this:
+
+|image0|
+
+This means our new observation facility module has been successfully
+loaded.
+
+BaseRoboticObservationFacility and BaseRoboticObservationForm
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You will have noticed our module consists of two classes that inherit
+from two other classes.
+
+``MyObservationFacility`` is the class that will contain the “business
+logic” for interacting with the remote observatory. This includes
+methods to submit observations, check observation status, etc. It
+inherits from ``BaseRoboticObservationFacility``, which contains some
+functionality that all observation facility classes will want.
+
+``MyObservationFacilityForm`` is the class that will display a GUI form
+for our users to create an observation. We can submit observations
+programmatically, but it is also nice to have a GUI for our users to
+use. The ``BaseRoboticObservationForm`` class, just like the previous
+super class, contains logic and layout that all observation facility
+form classes should contain.
+
+Implementing observation submission
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+Try to click on the button for ``MyFacility``. It should return an error
+that says everything it’s missing:
+
+::
+
+ Can't instantiate abstract class MyObservationFacility with abstract methods
+ data_products, get_form, get_observation_status, get_observation_url, get_observing_sites,
+ get_terminal_observing_states, submit_observation, validate_observation
+
+To start, let’s define new functions in ``MyObservationFacility`` for
+each missing function like so:
+
+.. code:: python
+
+ class MyObservationFacility(BaseRoboticObservationFacility):
+ name = 'MyFacility'
+ observation_types = [('OBSERVATION', 'Custom Observation')]
+
+ def data_products(self):
+ return
+
+ def get_form(self):
+ return
+ ...
+
+Reload the server, click the ``MyFacility`` button, and you should get .
+. . a different error! Progress!
+
+::
+
+ get_form() takes 1 positional argument but 2 were given
+
+To fix up ``get_form``, adjust it to:
+
+.. code:: python
+
+ def get_form(self, observation_type):
+ return MyObservationFacilityForm
+
+Reload the page and now it should look something like this:
+
+|image1|
+
+Some notes: 1. The form is empty, but we’ll fix that next. 2. The
+``name`` variable of ``MyObservationFacility`` determines what the top
+of the page says (``Submit an observation to MyFacility``). It also
+determines the name of the button under “Observe” on the target’s page.
+3. You should see a tab for ``Custom Observation`` as the only option on
+the page. This is read from the ``observation_types`` variable in
+``MyObservationFacility``. That variable is a list of 2-tuples. The
+second value of each tuple is what will be displayed on the webpage, as
+different tabs of observation types to submit. The first value of each
+tuple is what should be used to distinguish different observation types
+in your code. To see a demonstration of this, check out the `Las Cumbres
+Observatory `__
+facility’s ``observation_types`` and ``get_form``.
+
+Now let’s populate the form. Let’s assume our observatory only requires
+us to send 2 parameters (besides the target data): exposure_time and
+exposure_count. Let’s start by adding them to our form class:
+
+.. code:: python
+
+ from django import forms
+ from tom_observations.facility import GenericObservationFacility, GenericObservationForm
+
+
+ class MyObservationFacilityForm(GenericObservationForm):
+ exposure_time = forms.IntegerField()
+ exposure_count = forms.IntegerField()
+
+Notice that we’ve added the two field definitions on our form. We’ve
+also imported the django form module with ``from django import forms``.
+
+Now if we reload the page, we should see something like this:
+
+|image2|
+
+This is progress, but remember that most of the functions in
+``MyObservationFacility`` have blank return statements. Next we’ll
+implement the methods that perform actions with our form when we submit
+the observation request:
+
+.. code:: python
+
+ from django import forms
+ from tom_observations.facility import BaseRoboticObservationFacility, BaseRoboticObservationForm
+
+ class MyObservationFacilityForm(BaseRoboticObservationForm):
+ exposure_time = forms.IntegerField()
+ exposure_count = forms.IntegerField()
+
+ class MyObservationFacility(BaseRoboticObservationFacility):
+ name = 'MyFacility'
+ observation_types = [('OBSERVATION', 'Custom Observation')]
+
+ def data_products(self, observation_id, product_id=None):
+ return []
+
+ def get_form(self, observation_type):
+ return MyObservationFacilityForm
+
+ def get_observation_status(self, observation_id):
+ return ['IN_PROGRESS']
+
+ def get_observation_url(self, observation_id):
+ return ''
+
+ def get_observing_sites(self):
+ return {}
+
+ def get_terminal_observing_states(self):
+ return ['IN_PROGRESS', 'COMPLETED']
+
+ def submit_observation(self, observation_payload):
+ print(observation_payload)
+ return [1]
+
+ def validate_observation(self, observation_payload):
+ pass
+
+The important method here is ``submit_observation``. This method, when
+implemented fully, will send the observation payload to the remote
+observatory and then return a list of observation ids. Those ids will be
+stored in the database to be used later, in methods like
+``get_observation_status(self, observation_id)``. In our dummy
+implementation, we simply print out the observation payload and return a
+single fake id with ``return [1]``.
+
+If you now “submit” an observation using the MyFacility module, you
+should see this in the server console:
+
+::
+
+ {'target_id': 1, 'params': '{"facility": "MyFacility", "target_id": 1, "observation_type": "(\'OBSERVATION\', \'Custom Observation\')", "exposure_time": 100, "exposure_count": 2}'}
+
+That was our print statement! Additionally, you should see
+``1 upcoming observation`` on the target’s page, and if you navigate to
+its “Observations” tab you can see the parameters of the observation you
+just submitted in more detail.
+
+Filling in the rest of the functionality
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You’ll notice we added many more methods other than
+``submit_observation`` to our Facility class. For now they return dummy
+data, but when you adapt it to work with a real observatory you should
+fill them in with the correct logic so that the whole module works
+correctly with the TOM. You can view explanations of each method `in the
+source
+code `__
+
+###Airmass plotting for new facilities The last step in adding a new
+facility is to get it to appear on airmass plots. If you input two dates
+into the “Plan” form under the “Observe” tab on a target’s page, you’ll
+see the target’s visibility. By default, the plot shows you the airmass
+at LCO and Gemini sites.
+
+In our ``MyObservationFacility`` class, let’s define a new variable
+called ``SITES``. Modeling our ``SITES`` on the one defined for `Las
+Cumbres
+Observatory `__,
+we can easily put new sites into the airmass plots:
+
+.. code:: python
+
+ class MyObservationFacility(BaseRoboticObservationFacility):
+ name = 'MyFacility'
+ observation_types = [('OBSERVATION', 'Custom Observation')]
+
+ SITES = {
+ 'Itagaki': {
+ 'latitude': 38.188020,
+ 'longitude': 140.335113,
+ 'elevation': 350
+ }
+ }
+
+ ...
+
+ def get_observing_sites(self):
+ return self.SITES
+
+(Koichi Itagaki is an “amateur” astronomer in Japan who has discovered
+many extremely interesting supernovae.)
+
+Now the new observatory site should show up when you generate airmass
+plots. Even if the facilities you observe at are not API-accessible, you
+can still add them to your TOM’s airmass plots to judge what targets to
+observe when.
+
+Happy developing!
+
+Creating a custom manual facility
+---------------------------------
+
+.. |image0| image:: /_static/observation_module/myfacility.png
+.. |image1| image:: /_static/observation_module/empty_form.png
+.. |image2| image:: /_static/observation_module/fields.png
diff --git a/docs/observing/strategies.rst b/docs/observing/strategies.rst
new file mode 100644
index 000000000..49db1b572
--- /dev/null
+++ b/docs/observing/strategies.rst
@@ -0,0 +1,257 @@
+Dynamic Cadences and Observation Templates
+================================
+
+The TOM has a couple of unique concepts that may be unfamiliar to some
+at first, that will be describe here before going into detail.
+
+The first concept is that of an observation template. If an observer is consistently
+submitting observations with a lot of similar parameters, it may be
+useful to save those as a kind of template, which can just be loaded
+later. The TOM Toolkit offers an interface that allows facilities to
+define a template form, that will be saved as an observation template. The
+template can then be applied to an observation, with the remaining
+parameters filled in or changed. An observation template can also be
+creating from a past observation, with a button to do so that’s
+available on any ObservationRecord detail page.
+
+The second concept referred to is a dynamic cadence. A cadence is as it
+sounds–a series of observations that are performed at regular intervals.
+However, most observatories don’t have built-in support for cadences,
+and, if they do, they may be limited to a predetermined cadence. The TOM
+Toolkit, on the other hand, allows for a *dynamic* cadence. Because
+data is collected programmatically, and observations are submitted
+programmatically, a user can write their own cadence strategy to submit
+observations depending on the success of a prior observation or the data
+collected from a prior observation.
+
+Writing a custom dynamic cadence
+---------------------------------
+
+Many of the TOM modules leverage a plugin architecture that enables you
+to write your own implementation, and the cadence strategy plugin is no
+different. If you’re familiar with the other modules, you’ve already
+seen examples of this in the
+:doc:``Writing an alert broker <../customization/create_broker>``,
+:doc:``Writing an observation module ``, and
+:doc:``Customizing data processing <../customization/customizing_data_processing>``
+tutorials.
+
+Create a cadence strategy file
+------------------------------
+
+First, you’ll need a file where you’ll put your custom cadence strategy.
+If you have a fresh TOM installation, you’ll have a directory structure
+that looks something like this:
+
+::
+
+ ├── data
+ ├── db.sqlite3
+ ├── manage.py
+ ├── mytom
+ │ ├── __init__.py
+ │ ├── settings.py
+ │ ├── urls.py
+ │ └── wsgi.py
+ ├── static
+ ├── templates
+ └── tmp
+
+We’ll create a new file called ``mycadence.py`` and place it next to
+``settings.py``. To get started, we’ll just put a small skeleton into
+our new file, so to begin with, it should look like this:
+
+.. code:: python
+
+ from tom_observations.cadence import CadenceStrategy
+
+ class MyCadenceStrategy(CadenceStrategy):
+ pass
+
+We also need to add the cadence strategy to ``settings.py`` so that our
+TOM knows that it exists:
+
+.. code:: python
+
+ TOM_CADENCE_STRATEGIES = [
+ 'tom_observations.cadence.RetryFailedObservationsStrategy',
+ 'tom_observations.cadence.ResumeCadenceAfterFailureStrategy',
+ 'mytom.mycadence.MyCadenceStrategy'
+ ]
+
+Add logic to the new cadence strategy
+-------------------------------------
+
+You may have noticed that our ``MyCadenceStrategy`` class inherits from
+``CadenceStrategy``. The ``CadenceStrategy`` interface only has one
+method, which is ``run()``. All of the logic for a ``CadenceStrategy``
+lives in the ``run()`` method. Rather than demonstrating the
+implementation of a new cadence strategy, this tutorial is going to walk
+through the business logic of a built-in cadence strategy. We’re going
+to review the ``ResumeCadenceAfterFailureStrategy``.
+
+It should also be worth mentioning at this point that the
+``CadenceStrategy`` constructor takes a ``dynamic_cadence``. The
+``dynamic_cadence`` is the association of the cadence strategy and the
+observation group that make up the cadence, and is created in the
+``ObservationCreateView`` when the first observation of a cadence is submitted.
+
+The ``ResumeCadenceAfterFailureStrategy`` is designed to ensure that,
+even after an observation fails, the cadence remains consistent. If, for
+example, you submit an observation with a cadence of three days, and the
+observation fails, the cadence should attempt to get the observation as
+soon as possible, and then resume observing once every three days.
+
+Let’s look at the strategy piece by piece.
+
+.. code:: python
+
+ last_obs = self.dynamic_cadence.observation_group.observation_records.order_by('-created').first()
+ facility = get_service_class(last_obs.facility)()
+ facility.update_observation_status(last_obs.observation_id)
+ last_obs.refresh_from_db()
+
+The first thing this strategy does is get a couple of pieces of
+information. First, from the observation group that the cadence consists
+of, the most recent observation is selected. The facility class for the
+facility that the cadence is submitting observations to is also
+instantiated. With these values, the status of the most recent cadence
+observation is updated, and the ``ObservationRecord`` object is
+refreshed.
+
+.. code:: python
+
+ start_keyword, end_keyword = facility.get_start_end_keywords()
+ observation_payload = last_obs.parameters_as_dict
+ new_observations = []
+
+These lines are, again, just more setup. Each facility has its own
+unique keywords representing the start and the end of the observation
+window, so we get those from the facility class. Then, we get the
+original observation parameters that were submitted to the facility, and
+we initialize a list for any new observations that will be submitted
+when the cadence is updated.
+
+.. code:: python
+
+ if not last_obs.terminal:
+ return
+ elif last_obs.failed:
+ # Submit next observation to be taken as soon as possible
+ window_length = parse(observation_payload[end_keyword]) - parse(observation_payload[start_keyword])
+ observation_payload[start_keyword] = datetime.now().isoformat()
+ observation_payload[end_keyword] = (parse(observation_payload[start_keyword]) + window_length).isoformat()
+ else:
+ # Advance window normally according to cadence parameters
+ observation_payload = self.advance_window(
+ observation_payload, start_keyword=start_keyword, end_keyword=end_keyword
+ )
+
+Here we have some logic for the three cases–either the most recent
+observation hasn’t happened yet, it failed, or it succeeded. If it
+hasn’t happened, then there’s nothing to do–we’ll check again later. If
+if failed, we want to submit it again to be taken immediately, so we get
+the original length of the observation window, and set our new
+observation payload to start immediately, and end such that the new
+window length is the same. Finally, if our observation succeeded, we
+update our new observation parameters to start 72 hours after the last
+observation, using a utility method that’s part of the
+``ResumeCadenceAfterFailureStrategy`` called ``advance_window``.
+
+.. code:: python
+
+ obs_type = last_obs.parameters_as_dict.get('observation_type')
+ form = facility.get_form(obs_type)(observation_payload)
+ form.is_valid()
+ observation_ids = facility.submit_observation(form.observation_payload())
+
+ for observation_id in observation_ids:
+ # Create Observation record
+ record = ObservationRecord.objects.create(
+ target=last_obs.target,
+ facility=facility.name,
+ parameters=json.dumps(observation_payload),
+ observation_id=observation_id
+ )
+ self.dynamic_cadence.observation_group.observation_records.add(record)
+ self.dynamic_cadence.observation_group.save()
+ new_observations.append(record)
+
+ for obsr in new_observations:
+ facility = get_service_class(obsr.facility)()
+ facility.update_observation_status(obsr.observation_id)
+
+ return new_observations
+
+The last part of our strategy is when we submit our new observations.
+Regardless of how we modified the observing window, we initialize our
+observation form, validate it, and submit the observation to our
+facility. The rest of the code is saving any resulting observations to
+the database, getting their new status from the facility, and returning
+them.
+
+Just to review, here is the strategy’s ``run()`` in its entirety:
+
+.. code:: python
+
+ def run(self):
+ last_obs = self.dynamic_cadence.observation_group.observation_records.order_by('-created').first()
+ facility = get_service_class(last_obs.facility)()
+ facility.update_observation_status(last_obs.observation_id)
+ last_obs.refresh_from_db()
+ start_keyword, end_keyword = facility.get_start_end_keywords()
+ observation_payload = last_obs.parameters_as_dict
+ new_observations = []
+ if not last_obs.terminal:
+ return
+ elif last_obs.failed:
+ # Submit next observation to be taken as soon as possible
+ window_length = parse(observation_payload[end_keyword]) - parse(observation_payload[start_keyword])
+ observation_payload[start_keyword] = datetime.now().isoformat()
+ observation_payload[end_keyword] = (parse(observation_payload[start_keyword]) + window_length).isoformat()
+ else:
+ # Advance window normally according to cadence parameters
+ observation_payload = self.advance_window(
+ observation_payload, start_keyword=start_keyword, end_keyword=end_keyword
+ )
+
+ obs_type = last_obs.parameters_as_dict.get('observation_type')
+ form = facility.get_form(obs_type)(observation_payload)
+ form.is_valid()
+ observation_ids = facility.submit_observation(form.observation_payload())
+
+ for observation_id in observation_ids:
+ # Create Observation record
+ record = ObservationRecord.objects.create(
+ target=last_obs.target,
+ facility=facility.name,
+ parameters=json.dumps(observation_payload),
+ observation_id=observation_id
+ )
+ self.dynamic_cadence.observation_group.observation_records.add(record)
+ self.dynamic_cadence.observation_group.save()
+ new_observations.append(record)
+
+ for obsr in new_observations:
+ facility = get_service_class(obsr.facility)()
+ facility.update_observation_status(obsr.observation_id)
+
+ return new_observations
+
+Configuring the cadence strategy to run automatically
+-----------------------------------------------------
+
+As you may have noticed, the cadence strategies act on updates to the
+status of an ``ObservationRecord``. Ideally, we want the cadence
+strategies to run as soon as an observation status changes–so, we need
+to automate that and have it run periodically.
+
+Fortunately, the TOM Toolkit comes with a built-in management command to
+update all cadences in the TOM. If you’ve perused the TOM Toolkit
+documentation previously, you may have noticed a section about
+automation of tasks, and, more specifically, a subsection about
+:doc:`Using cron with a management command <../code/automation>`.
+You can simply apply the instructions here, but use the management
+command ``runcadencestrategies.py`` in place of the example. If you set
+your cron to run every few minutes or so, you’ll ensure that your
+cadences are kept up to date!
diff --git a/docs/requirements.txt b/docs/requirements.txt
index db6f1a037..e9077fea9 100644
--- a/docs/requirements.txt
+++ b/docs/requirements.txt
@@ -9,6 +9,7 @@ django-extensions
django-filter
django-gravatar2
django-guardian
+djangorestframework
fits2image
numpy
plotly
@@ -17,4 +18,6 @@ python-dateutil
recommonmark
requests
specutils
-sphinx>=2.1.2
\ No newline at end of file
+sphinx>=2.1.2
+tom_antares
+tom_scimma
\ No newline at end of file
diff --git a/docs/support.md b/docs/support.md
deleted file mode 100644
index a2fb3d5a1..000000000
--- a/docs/support.md
+++ /dev/null
@@ -1,21 +0,0 @@
-Getting Support
----
-
-This page will go over the process for reporting issues, requesting features, and getting
-support for the TOM Toolkit.
-
-### Reporting Issues
-
-Issue reporting can be done via the [Github issues page](https://github.com/TOMToolkit/tom_base/issues)
-of the tom_base project. Reporting an issue requires a Github account, but provides an easy way for
-developers to ask follow-up questions about an issue in order to resolve it.
-
-Please include as much detail as possible, as well as the steps taken that trigger the issue, and be sure to tag it with the "bug" tag!
-
-### Requesting Features
-
-Like issues, feature and enhancement requests should be done via the same [Github issues page](https://github.com/TOMToolkit/tom_base/issues) of the tom_base project. This is also a great place to see what's being worked on and what's already been requested, which will allow you to voice support for a backlogged feature to be reprioritized.
-
-### Support
-
-If you're looking for help with some aspect of your TOM, the [Github issues page](https://github.com/TOMToolkit/tom_base/issues) is once again the place to go. The "question" or "help wanted" tags should be very useful when looking for support, and the TOM Toolkit developers are more than happy to provide the help necessary to get your TOM running. You may also want to peruse the [Closed Issues](https://github.com/TOMToolkit/tom_base/issues?q=is%3Aissue+is%3Aclosed), where someone may have already had (and solved!) your problem.
\ No newline at end of file
diff --git a/docs/targets/index.rst b/docs/targets/index.rst
new file mode 100644
index 000000000..5947b4b52
--- /dev/null
+++ b/docs/targets/index.rst
@@ -0,0 +1,23 @@
+Targets
+=======
+
+.. toctree::
+ :maxdepth: 2
+ :hidden:
+
+ target_fields
+ ../api/tom_targets/models
+ ../api/tom_targets/views
+
+
+The ``Target``, along with the associated ``TargetList``, ``TargetExtra``, and ``TargetName``, are the core models of the
+TOM Toolkit. The ``Target`` defines the concept of an astronomical target.
+
+:doc:`Adding Custom Target Fields ` - Learn how to add custom fields to your TOM Targets if the
+defaults do not suffice.
+
+:doc:`Target API <../api/tom_targets/models>` - Take a look at the available properties for a ``Target`` and its associated models.
+
+:doc:`Target Views <../api/tom_targets/views>` - Familiarize yourself with the available Target Views.
+
+:doc:`Target Groups <../api/tom_targets/groups>` - Check out the functions for operating on Target Groups.
diff --git a/docs/targets/target_fields.rst b/docs/targets/target_fields.rst
new file mode 100644
index 000000000..59c416194
--- /dev/null
+++ b/docs/targets/target_fields.rst
@@ -0,0 +1,149 @@
+Adding Custom Fields to Targets
+-------------------------------
+
+Sometimes you’d like to store data for targets but the predefined fields
+that the TOM Toolkit provides aren’t enough. The TOM Toolkit allows you
+to define extra fields for your targets so you can associate different
+kinds of data with them. For example, you might be studying high
+redshift galaxies. In this case, it would make sense to be able to store
+the redshift of your targets. You could then do a search for targets
+with a redshift less than or greater than a particular value, or use the
+redshift value to make decisions in your science code.
+
+**Note**: There is a performance hit when using extra fields. Try to use
+the built in fields whenever possible.
+
+Enabling extra fields
+~~~~~~~~~~~~~~~~~~~~~
+
+To start, find the ``EXTRA_FIELDS`` definition in your ``settings.py``:
+
+.. code:: python
+
+ # Define extra target fields here. Types can be any of "number", "string", "boolean" or "datetime"
+ # For example:
+ # EXTRA_FIELDS = [
+ # {'name': 'redshift', 'type': 'number'},
+ # {'name': 'discoverer', 'type': 'string'}
+ # {'name': 'eligible', 'type': 'boolean'},
+ # {'name': 'dicovery_date', 'type': 'datetime'}
+ # ]
+ EXTRA_FIELDS = []
+
+We can define any number of extra fields in the array. Each item in the
+array is a dictionary with two values: name and type. Name is simply
+what you would like to name your field. Type is the datatype of the
+field and can be one of: ``number``, ``string``, ``boolean`` or
+``datetime``. These types allow the TOM Toolkit to properly store,
+filter and display these values elsewhere.
+
+As an example, let’s change the setting to look like this:
+
+.. code:: python
+
+ EXTRA_FIELDS = [
+ {'name': 'redshift', 'type': 'number'},
+ ]
+
+This will make an extra field with the name “redshift” and a type of
+“number” available to add to our targets.
+
+Using extra fields
+~~~~~~~~~~~~~~~~~~
+
+Now if you go to the target creation page, you should see the new field
+available:
+
+|image0|
+
+And if we go to our list of targets, we should see redshift as a field
+available to filter on:
+
+|image1|
+
+Extra fields with the ``number`` type allow filtering on range of
+values. The same goes for fields with the ``datetime`` type. ``string``
+types to a case insensitive inclusive search, and ``boolean`` fields to
+a simple matching comparison.
+
+Of course, redshift does appear on our target’s display page as well:
+
+|image2|
+
+To hide extra fields from the target page, we can set the “hidden” key
+(this doesn’t affect filtering and searching):
+
+.. code:: python
+
+ EXTRA_FIELDS = [
+ {'name': 'redshift', 'type': 'number', 'hidden': True},
+ ]
+
+And we can set a default value for an extra field by including a default
+key/value pair:
+
+.. code:: python
+
+ EXTRA_FIELDS = [
+ {'name': 'redshift', 'type': 'number', 'default': 0},
+ ]
+
+Displaying extra fields in templates
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If we want to display the redshift in other places, we can use a
+template filter to do that. For example, we might want to display the
+redshift value in the target list table.
+
+At the top of our template make sure to load ``targets_extras``:
+
+::
+
+ {% raw %}
+ {% load targets_extras %}
+ {% endraw %}
+
+Now we can use the ``target_extra_field`` filter wherever a target
+object is available in the template context:
+
+::
+
+ {% raw %}
+ {{ target|target_extra_field:"redshift" }}
+ {% endraw %}
+
+The result is the redshift value being printed on the template:
+
+|image3|
+
+Working with extra fields programatically
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+If you’d like to update or save extra fields to your targets in code,
+there are a few methods you can use. The simplest is to simply pass in a
+dictionary of extra data to your target’s ``save()`` method using the
+``extras`` keyword argument:
+
+.. code:: python
+
+ target = Target.objects.get(name='example')
+ target.save(extras={'foo': 42})
+
+The example target above will now have an extra field “foo” with the
+value 42.
+
+For more precise control, you can access ``TargetExtra`` models
+directly. To remove an extra, for example:
+
+.. code:: python
+
+ target = Target.objects.get(name='example')
+ target_extra = target.targetextra_set.get(key='foo')
+ target_extra.delete()
+
+The above deleted the target extra on a target with the key of “foo”.
+
+.. |image0| image:: /_static/target_fields_doc/redshift.png
+.. |image1| image:: /_static/target_fields_doc/redshift_filter.png
+.. |image2| image:: /_static/target_fields_doc/redshift_display.png
+.. |image3| image:: /_static/target_fields_doc/redshift_tag.png
diff --git a/releasenotes.md b/releasenotes.md
new file mode 100644
index 000000000..44453b7f7
--- /dev/null
+++ b/releasenotes.md
@@ -0,0 +1,50 @@
+# Release Notes
+
+## 2.0.0
+
+- Renamed `ALERT_CREDENTIALS` and `BROKER_CREDENTIALS` to `BROKERS` as a catchall for any broker-specific values.
+- Added support for custom `CadenceStrategy` layouts.
+- Moved settings for `TNSHarvester` into `settings.HARVESTERS` to maintain consistency.
+- Updated `tom_alerts.GenericBroker` interface to support submission upstream to a broker, if implemented.
+- Fixed `TNSBroker` to get the correct object name.
+- Added stub `SCIMMABroker`.
+- Removed `tom_publications` from `tom_base`, and placed it in a separate `tom_publications` repository.
+- Upgraded a number of dependencies, including `astroplan`, `astropy`, and multiple `django`-related libraries.
+- Added tests for `lco.py`, `soar.py`, `alerce.py`, and `mars.py`.
+- Added canary tests for `mars.py` and `alerce.py`.
+
+### Breaking changes
+
+- Migrations are required for this version.
+- Due to the renaming of `BROKER_CREDENTIALS` and `ALERT_CREDENTIALS` to `BROKERS`, TOM Toolkit users will need to consolidate their broker configurations in `settings.py` into the `BROKERS` dict.
+- Because the built-in cadence strategies were moved into their own files, users of the cadence strategies will need to update their `settings.TOM_CADENCE_STRATEGIES` to include the values as seen in this commit: https://github.com/TOMToolkit/tom_base/blob/82101a92a9c19f0ff8ab0f59ecb758bc47824252/tom_base/settings.py#L214
+- Users of the `TNSHarvester` will need to introduce a dict in `settings` called `HARVESTERS` with a sub-dict `TNS` to store the relevant `api_key`.
+- Due to the removal of `tom_publications`, TOM Toolkit users will need to either add `tom_publications` to their dependencies, or:
+ - Remove `tom_publications` from `INSTALLED_APPS`.
+ - Remove `publications_extras` from the following templates, if they've been customized: `observation_groups.html`, `target_grouping.html`.
+ - Remove references to `latex_button_group` from the templates referenced above, if they've been customized.
+- The `LCOBaseForm` methods `instrument_choices`, `instrument_to_type`, and `filter_choices` were re-implemented as static methods, and any subclasses will need to add a `staticmethod` decorator, modify the method signature, and replace calls to `self` within the method to calls to the class name.
+
+## 1.6.1
+
+ - This release pins the Django version in order to address a security vulnerability.
+
+### What to watch out for
+
+ - The Django version is now pinned at 3.0.7, where previously it allowed >=2.2. You'll need to ensure that any custom code is compatible with Django >=3.0.7.
+
+## 1.6.0
+
+ - New methods expand the Facility API to support reporting Facility status and weather: `get_facility_status()` and `get_facility_weather_url()`. When these methods are implemented by a Facility provider, this information can be made available in your TOM.
+ - A new template tag, `facility_status()`, is available to present this information.
+
+## 1.5.0
+
+ - Introduced a manual facility interface for classical observing.
+ - Introduced a view and corresponding form to add existing API-based observations to a Target.
+ - Introduced a view and corresponding form to update an existing manual observation with an API-based observation ID.
+
+
+### What to watch out for
+
+ - For facility implementers: in order to support a Manual Facility Interface, the team created a `BaseObservationFacility` and two abstract implementations of it, `BaseRoboticObservationFacility` and `BaseManualObservationFacility`. `BaseRoboticObservationFacility` was aliased as `GenericObservationFacility` to support backwards compatibility, but will be removed in 2.0.
\ No newline at end of file
diff --git a/requirements.txt b/requirements.txt
index a4af5aa10..3595830b0 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1 +1,2 @@
-e .[test]
+# see setup.py install_requires for list
diff --git a/setup.py b/setup.py
index 0de499e21..f36713614 100644
--- a/setup.py
+++ b/setup.py
@@ -7,13 +7,12 @@
setup(
name='tomtoolkit',
- version='1.4.0',
description='The TOM Toolkit and base modules',
long_description=long_description,
long_description_content_type='text/markdown',
url='https://tomtoolkit.github.io',
author='TOM Toolkit Project',
- author_email='ariba@lco.global',
+ author_email='dcollom@lco.global',
classifiers=[
'Development Status :: 3 - Alpha',
'Intended Audience :: Science/Research',
@@ -26,30 +25,34 @@
],
keywords=['tomtoolkit', 'astronomy', 'astrophysics', 'cosmology', 'science', 'fits', 'observatory'],
packages=find_packages(),
+ use_scm_version=True,
+ setup_requires=['setuptools_scm', 'wheel'],
install_requires=[
- 'django>=2.2', # TOM Toolkit requires db math functions
- 'django-bootstrap4',
- 'django-extensions',
- 'django-filter',
- 'django-contrib-comments>=1.9.2', # Earlier version are incompatible with Django >= 3.0
- 'django-gravatar2',
- 'django-crispy-forms',
- 'django-guardian',
- 'numpy',
- 'python-dateutil',
- 'requests',
- 'astroquery',
- 'astropy==4.0',
- 'astroplan',
- 'plotly',
- 'matplotlib',
- 'pillow',
- 'fits2image',
- 'specutils==0.7',
+ 'astroquery==0.4.1',
+ 'astroplan==0.7',
+ 'astropy==4.1',
+ 'beautifulsoup4==4.9.3',
'dataclasses; python_version < "3.7"',
+ 'django==3.1.3', # TOM Toolkit requires db math functions
+ 'djangorestframework==3.12.2',
+ 'django-bootstrap4==2.3.1',
+ 'django-contrib-comments==1.9.2', # Earlier version are incompatible with Django >= 3.0
+ 'django-crispy-forms==1.9.2',
+ 'django-extensions==3.0.9',
+ 'django-gravatar2==1.4.4',
+ 'django-filter==2.4.0',
+ 'django-guardian==2.3.0',
+ 'fits2image==0.4.3',
+ 'Markdown==3.3.3', # django-rest-framework doc headers require this to support Markdown
+ 'numpy==1.19.4',
+ 'pillow==8.0.1',
+ 'plotly==4.12.0',
+ 'python-dateutil==2.8.1',
+ 'requests==2.25.0',
+ 'specutils==1.1',
],
extras_require={
- 'test': ['factory_boy']
+ 'test': ['factory_boy==3.1.0']
},
include_package_data=True,
)
diff --git a/tom_alerts/alerts.py b/tom_alerts/alerts.py
index 38085cac3..35f8a1902 100644
--- a/tom_alerts/alerts.py
+++ b/tom_alerts/alerts.py
@@ -1,14 +1,18 @@
-from django.conf import settings
-from django import forms
-from importlib import import_module
-from datetime import datetime
+from abc import ABC, abstractmethod
from dataclasses import dataclass
-from crispy_forms.helper import FormHelper
-from crispy_forms.layout import Submit, Layout
+from datetime import datetime
+from importlib import import_module
import json
-from abc import ABC, abstractmethod
+
+from django import forms
+from django.conf import settings
+from django.shortcuts import reverse
+from crispy_forms.bootstrap import StrictButton
+from crispy_forms.helper import FormHelper
+from crispy_forms.layout import Layout, Submit
from tom_alerts.models import BrokerQuery
+from tom_observations.models import ObservationRecord
from tom_targets.models import Target
@@ -18,7 +22,8 @@
'tom_alerts.brokers.scout.ScoutBroker',
'tom_alerts.brokers.alerce.ALeRCEBroker',
'tom_alerts.brokers.antares.ANTARESBroker',
- 'tom_alerts.brokers.gaia.GaiaBroker'
+ 'tom_alerts.brokers.gaia.GaiaBroker',
+ 'tom_alerts.brokers.scimma.SCIMMABroker',
]
@@ -81,17 +86,24 @@ class GenericAlert:
def to_target(self):
"""
- Returns a Target instance for an object defined by an alert.
+ Returns a Target instance for an object defined by an alert, as well as
+ any TargetExtra or additional TargetNames.
:returns: representation of object for an alert
:rtype: `Target`
+
+ :returns: dict of extras to be added to the new Target
+ :rtype: `dict`
+
+ :returns: list of aliases to be added to the new Target
+ :rtype: `list`
"""
return Target(
name=self.name,
type='SIDEREAL',
ra=self.ra,
dec=self.dec
- )
+ ), {}, []
class GenericQueryForm(forms.Form):
@@ -139,6 +151,32 @@ def save(self, query_id=None):
return query
+class GenericUpstreamSubmissionForm(forms.Form):
+ target = forms.ModelChoiceField(required=False, queryset=Target.objects.all(), widget=forms.HiddenInput())
+ observation_record = forms.ModelChoiceField(required=False, queryset=ObservationRecord.objects.all(),
+ widget=forms.HiddenInput())
+ redirect_url = forms.CharField(required=False, max_length=100, widget=forms.HiddenInput())
+
+ def __init__(self, *args, **kwargs):
+ broker_name = kwargs.pop('broker') # NOTE: parent constructor is not expecting broker and will fail
+ super().__init__(*args, **kwargs)
+ self.helper = FormHelper()
+ self.helper.form_action = reverse('tom_alerts:submit-alert', kwargs={'broker': broker_name})
+ self.helper.layout = Layout(
+ 'target',
+ 'observation_record',
+ 'redirect_url',
+ StrictButton(f'Submit to {broker_name}', type='submit', css_class='btn-outline-primary'))
+
+ def clean(self):
+ cleaned_data = super().clean()
+
+ if not (cleaned_data.get('target') or cleaned_data.get('observation_record')):
+ raise forms.ValidationError('Must provide either Target or ObservationRecord to be submitted upstream.')
+
+ return cleaned_data
+
+
class GenericBroker(ABC):
"""
The ``GenericBroker`` provides an interface for implementing a broker module. It contains a number of methods to be
@@ -146,8 +184,9 @@ class GenericBroker(ABC):
make use of a broker module, add the path to ``TOM_ALERT_CLASSES`` in your ``settings.py``.
For an implementation example, please see
- https://github.com/TOMToolkit/tom_base/blob/master/tom_alerts/brokers/mars.py
+ https://github.com/TOMToolkit/tom_base/blob/main/tom_alerts/brokers/mars.py
"""
+ alert_submission_form = GenericUpstreamSubmissionForm
@abstractmethod
def fetch_alerts(self, parameters):
@@ -159,7 +198,6 @@ def fetch_alerts(self, parameters):
:param parameters: JSON string of query parameters
:type parameters: str
"""
- pass
def fetch_alert(self, id):
"""
@@ -193,6 +231,22 @@ def to_target(self, alert):
"""
pass
+ def submit_upstream_alert(self, target=None, observation_record=None, **kwargs):
+ """
+ Submits an alert upstream back to the broker. At least one of a target or an
+ observation record must be provided.
+
+ :param target: ``Target`` object to be converted to an alert and submitted upstream
+ :type target: ``Target``
+
+ :param observation_record: ``ObservationRecord`` object to be converted to an alert and submitted upstream
+ :type observation_record: ``ObservationRecord``
+
+ :returns: True or False depending on success of message submission
+ :rtype: bool
+ """
+ pass
+
@abstractmethod
def to_generic_alert(self, alert):
"""
diff --git a/tom_alerts/brokers/alerce.py b/tom_alerts/brokers/alerce.py
index 57200398b..461bed072 100644
--- a/tom_alerts/brokers/alerce.py
+++ b/tom_alerts/brokers/alerce.py
@@ -1,9 +1,10 @@
+from datetime import datetime, timedelta
import requests
-from django import forms
-from crispy_forms.layout import Layout, Div, Fieldset
from astropy.time import Time, TimezoneInfo
-import datetime
+from crispy_forms.layout import Layout, Div, Fieldset
+from django import forms
+from django.core.cache import cache
from tom_alerts.alerts import GenericQueryForm, GenericBroker, GenericAlert
from tom_targets.models import Target
@@ -12,10 +13,10 @@
ALERCE_SEARCH_URL = 'https://ztf.alerce.online/query'
ALERCE_CLASSES_URL = 'https://ztf.alerce.online/get_current_classes'
-SORT_CHOICES = [("nobs", "Number Of Epochs"),
- ("lastmjd", "Last Detection"),
- ("pclassrf", "Late Probability"),
- ("pclassearly", "Early Probability")]
+SORT_CHOICES = [('nobs', 'Number Of Epochs'),
+ ('lastmjd', 'Last Detection'),
+ ('pclassrf', 'Late Probability'),
+ ('pclassearly', 'Early Probability')]
PAGES_CHOICES = [
(i, i) for i in [1, 5, 10, 15]
@@ -28,9 +29,6 @@
class ALeRCEQueryForm(GenericQueryForm):
- RF_CLASSIFIERS = []
- STAMP_CLASSIFIERS = []
-
nobs__gt = forms.IntegerField(
required=False,
label='Detections Lower',
@@ -41,19 +39,21 @@ class ALeRCEQueryForm(GenericQueryForm):
label='Detections Upper',
widget=forms.TextInput(attrs={'placeholder': 'Max number of epochs'})
)
- classrf = forms.ChoiceField(
+ classrf = forms.TypedChoiceField(
required=False,
label='Late Classifier (Random Forest)',
- choices=RF_CLASSIFIERS
+ choices=[], # Choices are populated dynamically in the constructor
+ coerce=int
)
pclassrf = forms.FloatField(
required=False,
label='Classifier Probability (Random Forest)'
)
- classearly = forms.ChoiceField(
+ classearly = forms.TypedChoiceField(
required=False,
label='Early Classifier (Stamp Classifier)',
- choices=STAMP_CLASSIFIERS
+ choices=[], # Choices are populated dynamically in the constructor
+ coerce=int
)
pclassearly = forms.FloatField(
required=False,
@@ -77,17 +77,20 @@ class ALeRCEQueryForm(GenericQueryForm):
mjd__gt = forms.FloatField(
required=False,
label='Min date of first detection ',
- widget=forms.TextInput(attrs={'placeholder': 'Date (MJD)'})
+ widget=forms.TextInput(attrs={'placeholder': 'Date (MJD)'}),
+ min_value=0.0
)
mjd__lt = forms.FloatField(
required=False,
label='Max date of first detection',
- widget=forms.TextInput(attrs={'placeholder': 'Date (MJD)'})
+ widget=forms.TextInput(attrs={'placeholder': 'Date (MJD)'}),
+ min_value=0.0
)
relative_mjd__gt = forms.FloatField(
required=False,
label='Relative date of object discovery.',
- widget=forms.TextInput(attrs={'placeholder': 'Hours'})
+ widget=forms.TextInput(attrs={'placeholder': 'Hours'}),
+ min_value=0.0
)
sort_by = forms.ChoiceField(
choices=SORT_CHOICES,
@@ -97,28 +100,21 @@ class ALeRCEQueryForm(GenericQueryForm):
max_pages = forms.TypedChoiceField(
choices=PAGES_CHOICES,
required=False,
- label='Max Number of Pages'
+ label='Max Number of Pages',
+ coerce=int
)
- records = forms.ChoiceField(
+ records = forms.TypedChoiceField(
choices=RECORDS_CHOICES,
required=False,
- label='Records per page'
+ label='Records per page',
+ coerce=int
)
def __init__(self, *args, **kwargs):
super().__init__(*args, **kwargs)
- response = requests.post(ALERCE_CLASSES_URL)
- response.raise_for_status()
- parsed = response.json()
-
- EARLY_CHOICES = [(c["id"], c["name"]) for c in parsed["early"]]
- EARLY_CHOICES.insert(0, (None, ""))
- LATE_CHOICES = [(c["id"], c["name"]) for c in parsed["late"]]
- LATE_CHOICES.insert(0, (None, ""))
-
- self.fields["classearly"].choices = EARLY_CHOICES
- self.fields["classrf"].choices = LATE_CHOICES
+ self.fields['classearly'].choices = self.early_classifier_choices()
+ self.fields['classrf'].choices = self.late_classifier_choices()
self.helper.layout = Layout(
self.common_layout,
@@ -133,7 +129,7 @@ def __init__(self, *args, **kwargs):
'nobs__lt',
css_class='col',
),
- css_class="form-row",
+ css_class='form-row',
)
),
Fieldset(
@@ -149,7 +145,7 @@ def __init__(self, *args, **kwargs):
'pclassearly',
css_class='col',
),
- css_class="form-row",
+ css_class='form-row',
)
),
Fieldset(
@@ -157,7 +153,7 @@ def __init__(self, *args, **kwargs):
Div(
Div(
'ra',
- css_class="col"
+ css_class='col'
),
Div(
'dec',
@@ -167,14 +163,14 @@ def __init__(self, *args, **kwargs):
'sr',
css_class='col'
),
- css_class="form-row"
+ css_class='form-row'
)
),
Fieldset(
'Time Filters',
Div(
Fieldset(
- "Relative time",
+ 'Relative time',
Div(
'relative_mjd__gt',
css_class='col',
@@ -182,7 +178,7 @@ def __init__(self, *args, **kwargs):
css_class='col'
),
Fieldset(
- "Absolute time",
+ 'Absolute time',
Div(
Div(
'mjd__gt',
@@ -192,115 +188,172 @@ def __init__(self, *args, **kwargs):
'mjd__lt',
css_class='col',
),
- css_class="form-row"
+ css_class='form-row'
)
),
- css_class="form-row"
+ css_class='form-row'
)
),
Fieldset(
'General Parameters',
Div(
Div(
- "sort_by",
- css_class="col"
+ 'sort_by',
+ css_class='col'
),
Div(
- "records",
- css_class="col"
+ 'records',
+ css_class='col'
),
Div(
- "max_pages",
- css_class="col"
+ 'max_pages',
+ css_class='col'
),
- css_class="form-row"
+ css_class='form-row'
)
),
)
+ @staticmethod
+ def _get_classifiers():
+ cached_classifiers = cache.get('alerce_classifiers')
+
+ if not cached_classifiers:
+ response = requests.get(ALERCE_CLASSES_URL)
+ response.raise_for_status()
+ cached_classifiers = response.json()
+
+ return cached_classifiers
+
+ def clean_sort_by(self):
+ return self.cleaned_data['sort_by'] if self.cleaned_data['sort_by'] else 'nobs'
+
+ def clean_records(self):
+ return self.cleaned_data['records'] if self.cleaned_data['records'] else 20
+
+ def clean_relative_mjd__gt(self):
+ if self.cleaned_data['relative_mjd__gt']:
+ return Time(datetime.now() - timedelta(hours=self.cleaned_data['relative_mjd__gt'])).mjd
+ return None
+
+ def clean(self):
+ cleaned_data = super().clean()
+
+ # Ensure that all cone search fields are present
+ if any(cleaned_data[k] for k in ['ra', 'dec', 'sr']) and not all(cleaned_data[k] for k in ['ra', 'dec', 'sr']):
+ raise forms.ValidationError('All of RA, Dec, and Search Radius must be included to execute a cone search.')
+
+ # Ensure that both relative and absolute time filters are not present
+ if any(cleaned_data[k] for k in ['mjd__lt', 'mjd__gt']) and cleaned_data.get('relative_mjd__gt'):
+ raise forms.ValidationError('Cannot filter by both relative and absolute time.')
+
+ # Ensure that absolute time filters have sensible values
+ if all(cleaned_data[k] for k in ['mjd__lt', 'mjd__gt']) and cleaned_data['mjd__lt'] <= cleaned_data['mjd__gt']:
+ raise forms.ValidationError('Min date of first detection must be earlier than max date of first detection.')
+
+ return cleaned_data
+
+ def early_classifier_choices(self):
+ return [(None, '')] + sorted([(c['id'], c['name']) for c in self._get_classifiers()['early']],
+ key=lambda classifier: classifier[1])
+
+ def late_classifier_choices(self):
+ return [(None, '')] + sorted([(c['id'], c['name']) for c in self._get_classifiers()['late']],
+ key=lambda classifier: classifier[1])
+
class ALeRCEBroker(GenericBroker):
name = 'ALeRCE'
form = ALeRCEQueryForm
- def _fetch_alerts_payload(self, parameters):
- payload = {
- 'page': parameters.get('page', 1),
- 'records_per_pages': int(parameters.get('records', 20)),
- 'sortBy': parameters.get('sort_by'),
- 'query_parameters': {
+ def _clean_coordinate_parameters(self, parameters):
+ if all([parameters['ra'], parameters['dec'], parameters['sr']]):
+ return {
+ 'ra': parameters['ra'],
+ 'dec': parameters['dec'],
+ 'sr': parameters['sr']
}
- }
- if parameters.get('total'):
- payload['total'] = parameters.get('total')
+ else:
+ return None
- if any([parameters['nobs__gt'],
- parameters['nobs__lt'],
- parameters['classrf'],
- parameters['pclassrf'],
- parameters['classearly'],
- parameters['pclassearly']]):
- filters = {}
- if any([parameters['nobs__gt'],
- parameters['nobs__lt']]):
- filters['nobs'] = {}
- if parameters['nobs__gt']:
- filters['nobs']['min'] = parameters['nobs__gt']
- if parameters['nobs__lt']:
- filters['nobs']['max'] = parameters['nobs__lt']
- if parameters['classrf']:
- filters['classrf'] = int(parameters['classrf'])
- if parameters['pclassrf']:
- filters['pclassrf'] = parameters['pclassrf']
- if parameters['classearly']:
- filters['classearly'] = int(parameters['classearly'])
- if parameters['pclassearly']:
- filters['pclassearly'] = parameters['pclassearly']
- payload['query_parameters']['filters'] = filters
-
- if all([parameters['ra'],
- parameters['dec'],
- parameters['sr']]):
- coordinates = {}
- if parameters['ra']:
- coordinates['ra'] = parameters['ra']
- if parameters['dec']:
- coordinates['dec'] = parameters['dec']
- if parameters['sr']:
- coordinates['sr'] = parameters['sr']
- payload['query_parameters']['coordinates'] = coordinates
+ def _clean_date_parameters(self, parameters):
+ dates = {}
- if any([parameters['mjd__gt'],
- parameters['mjd__lt'],
- parameters['relative_mjd__gt']]):
+ if any(parameters[k] for k in ['mjd__gt', 'mjd__lt']):
dates = {'firstmjd': {}}
if parameters['mjd__gt']:
dates['firstmjd']['min'] = parameters['mjd__gt']
- elif parameters['relative_mjd__gt']:
- now = datetime.datetime.utcnow()
- relative = now - datetime.timedelta(hours=parameters['relative_mjd__gt'])
- relative_astro = Time(relative)
- dates['firstmjd']['min'] = relative_astro.mjd
-
if parameters['mjd__lt']:
dates['firstmjd']['max'] = parameters['mjd__lt']
- payload['query_parameters']['dates'] = dates
+ elif parameters['relative_mjd__gt']:
+ dates = {'firstmjd': {'min': parameters['relative_mjd__gt']}}
+
+ return dates
+
+ def _clean_filter_parameters(self, parameters):
+ filters = {}
+
+ if any(parameters[k] is not None for k in ['nobs__gt', 'nobs__lt']):
+ filters['nobs'] = {}
+ if parameters['nobs__gt']:
+ filters['nobs']['min'] = parameters['nobs__gt']
+ if parameters['nobs__lt']:
+ filters['nobs']['max'] = parameters['nobs__lt']
+ filters.update({k: parameters[k]
+ for k in ['classrf', 'pclassrf', 'classearly', 'pclassearly']
+ if parameters[k]})
+
+ return filters
+
+ def _clean_parameters(self, parameters):
+ payload = {
+ 'page': parameters.get('page', 1),
+ 'records_per_pages': parameters.get('records', 20),
+ 'sortBy': parameters.get('sort_by', 'nobs'),
+ 'query_parameters': {}
+ }
+
+ if parameters.get('total'):
+ payload['total'] = parameters.get('total')
+
+ payload['query_parameters']['filters'] = self._clean_filter_parameters(parameters)
+
+ coordinates = self._clean_coordinate_parameters(parameters)
+ if coordinates:
+ payload['query_parameters']['coordinates'] = coordinates
+
+ payload['query_parameters']['dates'] = self._clean_date_parameters(parameters)
return payload
def fetch_alerts(self, parameters):
- payload = self._fetch_alerts_payload(parameters)
+ payload = self._clean_parameters(parameters)
response = requests.post(ALERCE_SEARCH_URL, json=payload)
response.raise_for_status()
parsed = response.json()
alerts = [alert_data for alert, alert_data in parsed['result'].items()]
- if parsed['page'] < parsed['num_pages'] and parsed['page'] != int(parameters["max_pages"]):
+ if parsed['page'] < parsed['num_pages'] and parsed['page'] != parameters['max_pages']:
parameters['page'] = parameters.get('page', 1) + 1
parameters['total'] = parsed.get('total')
alerts += self.fetch_alerts(parameters)
return iter(alerts)
def fetch_alert(self, id):
+ """
+ The response for a single alert is as follows:
+
+ {
+ "total": 1,
+ "num_pages": 1,
+ "page": 1,
+ "result": {
+ "ZTF20acnsdjd": {
+ "oid": "ZTF20acnsdjd",
+ other alert values
+ }
+ }
+ }
+ """
payload = {
'query_parameters': {
'filters': {
@@ -310,7 +363,7 @@ def fetch_alert(self, id):
}
response = requests.post(ALERCE_SEARCH_URL, json=payload)
response.raise_for_status()
- return response.json()['result'][0]
+ return list(response.json()['result'].items())[0][1]
def to_target(self, alert):
return Target.objects.create(
@@ -325,18 +378,20 @@ def to_generic_alert(self, alert):
timestamp = Time(alert['lastmjd'], format='mjd', scale='utc').to_datetime(timezone=TimezoneInfo())
else:
timestamp = ''
- url = '{0}/{1}/{2}'.format(ALERCE_URL, 'object', alert['oid'])
-
- exits = (alert['mean_magpsf_g'] is None and alert['mean_magpsf_r'] is not None)
- both_exists = (alert['mean_magpsf_g'] is not None and alert['mean_magpsf_r'] is not None)
- bigger = (both_exists and (alert['mean_magpsf_r'] < alert['mean_magpsf_g'] is not None))
- is_r = any([exits, bigger])
+ url = f'{ALERCE_URL}/object/{alert["oid"]}'
- max_mag = alert['mean_magpsf_r'] if is_r else alert['mean_magpsf_g']
+ # Use the smaller value between r and g if both are present, else use the value that is present
+ mag = None
+ if alert['mean_magpsf_r'] is not None and alert['mean_magpsf_g'] is not None:
+ mag = alert['mean_magpsf_g'] if alert['mean_magpsf_r'] > alert['mean_magpsf_g'] else alert['mean_magpsf_r']
+ elif alert['mean_magpsf_r'] is not None:
+ mag = alert['mean_magpsf_r']
+ elif alert['mean_magpsf_g'] is not None:
+ mag = alert['mean_magpsf_g']
- if alert['pclassrf']:
- score = alert["pclassrf"]
- elif alert['pclassearly']:
+ if alert['pclassrf'] is not None:
+ score = alert['pclassrf']
+ elif alert['pclassearly'] is not None:
score = alert['pclassearly']
else:
score = None
@@ -348,6 +403,6 @@ def to_generic_alert(self, alert):
name=alert['oid'],
ra=alert['meanra'],
dec=alert['meandec'],
- mag=max_mag,
+ mag=mag,
score=score
)
diff --git a/tom_alerts/brokers/antares.py b/tom_alerts/brokers/antares.py
index e9eda7cff..43199c817 100644
--- a/tom_alerts/brokers/antares.py
+++ b/tom_alerts/brokers/antares.py
@@ -1,6 +1,6 @@
from crispy_forms.layout import Layout, HTML
-from tom_alerts.alerts import GenericBroker, GenericQueryForm
+from tom_alerts.alerts import GenericBroker, GenericQueryForm, GenericAlert
class ANTARESQueryForm(GenericQueryForm):
@@ -21,3 +21,12 @@ def __init__(self, *args, **kwargs):
class ANTARESBroker(GenericBroker):
name = 'ANTARES'
form = ANTARESQueryForm
+
+ def fetch_alerts(self, parameters):
+ return iter([])
+
+ def process_reduced_data(self, target, alert=None):
+ return
+
+ def to_generic_alert(self, alert):
+ return GenericAlert()
diff --git a/tom_alerts/brokers/gaia.py b/tom_alerts/brokers/gaia.py
index 5f038bbb7..a7ea9e9e5 100644
--- a/tom_alerts/brokers/gaia.py
+++ b/tom_alerts/brokers/gaia.py
@@ -1,18 +1,19 @@
-from tom_alerts.alerts import GenericQueryForm, GenericAlert, GenericBroker
-from tom_alerts.models import BrokerQuery
-from tom_targets.models import Target
-from tom_dataproducts.models import ReducedDatum
from dateutil.parser import parse
-from django import forms
+import json
+import re
+import requests
+from requests.exceptions import HTTPError
+
from astropy.coordinates import SkyCoord
from astropy.time import Time, TimezoneInfo
import astropy.units as u
-import requests
-from requests.exceptions import HTTPError
-import json
-from os import path
+from bs4 import BeautifulSoup
+from django import forms
+
+from tom_alerts.alerts import GenericAlert, GenericBroker, GenericQueryForm
+from tom_dataproducts.models import ReducedDatum
-BROKER_URL = 'http://gsaweb.ast.cam.ac.uk/alerts/alertsindex'
+BASE_BROKER_URL = 'http://gsaweb.ast.cam.ac.uk'
class GaiaQueryForm(GenericQueryForm):
@@ -23,12 +24,20 @@ class GaiaQueryForm(GenericQueryForm):
help_text='RA,Dec,radius in degrees'
)
+ def clean_cone(self):
+ cone = self.cleaned_data['cone']
+ if cone:
+ cone_params = cone.split(',')
+ if len(cone_params) != 3:
+ raise forms.ValidationError('Cone search parameters must be in the format \'RA,Dec,Radius\'.')
+ return cone
+
def clean(self):
- if len(self.cleaned_data['target_name']) == 0 and \
- len(self.cleaned_data['cone']) == 0:
- raise forms.ValidationError(
- "Please enter either a target name or cone search parameters"
- )
+ super().clean()
+ if not (self.cleaned_data.get('target_name') or self.cleaned_data.get('cone')):
+ raise forms.ValidationError('Please enter either a target name or cone search parameters.')
+ elif self.cleaned_data.get('target_name') and self.cleaned_data.get('cone'):
+ raise forms.ValidationError('Please only enter one of target name or cone search parameters.')
class GaiaBroker(GenericBroker):
@@ -37,16 +46,21 @@ class GaiaBroker(GenericBroker):
def fetch_alerts(self, parameters):
"""Must return an iterator"""
- response = requests.get(BROKER_URL)
+ response = requests.get(f'{BASE_BROKER_URL}/alerts/alertsindex')
response.raise_for_status()
- html_data = response.text.split('\n')
- for line in html_data:
- if 'var alerts' in line:
- alerts_data = line.replace('var alerts = ', '')
- alerts_data = alerts_data.replace('\n', '').replace(';', '')
+ soup = BeautifulSoup(response.content, 'html.parser')
+ script_tags = soup.find_all('script')
+ alerts = None
+
+ alerts_pattern = re.compile(r'var alerts = \[(.*?)];')
+ for script in script_tags:
+ m = alerts_pattern.match(str(script.string).strip())
+ if m is not None:
+ alerts = '['+m.group(1)+']'
+ break
- alert_list = json.loads(alerts_data)
+ alert_list = json.loads(alerts)
if parameters['cone'] is not None and len(parameters['cone']) > 0:
cone_params = parameters['cone'].split(',')
@@ -58,8 +72,7 @@ def fetch_alerts(self, parameters):
frame="icrs", unit="deg")
filtered_alerts = []
- if parameters['target_name'] is not None and \
- len(parameters['target_name']) > 0:
+ if parameters.get('target_name'):
for alert in alert_list:
if parameters['target_name'] in alert['name']:
filtered_alerts.append(alert)
@@ -87,7 +100,8 @@ def fetch_alert(self, target_name):
def to_generic_alert(self, alert):
timestamp = parse(alert['obstime'])
- url = BROKER_URL.replace('/alerts/alertsindex', alert['per_alert']['link'])
+ alert_link = alert.get('per_alert', {})['link']
+ url = f'{BASE_BROKER_URL}/{alert_link}'
return GenericAlert(
timestamp=timestamp,
@@ -102,8 +116,6 @@ def to_generic_alert(self, alert):
def process_reduced_data(self, target, alert=None):
- base_url = BROKER_URL.replace('/alertsindex', '/alert')
-
if not alert:
try:
alert = self.fetch_alert(target.name)
@@ -111,13 +123,14 @@ def process_reduced_data(self, target, alert=None):
except HTTPError:
raise Exception('Unable to retrieve alert information from broker')
- alert_url = BROKER_URL.replace('/alerts/alertsindex',
- alert['per_alert']['link'])
-
- if alert:
- lc_url = path.join(base_url, alert['name'], 'lightcurve.csv')
+ if alert is not None:
+ alert_name = alert['name']
+ alert_link = alert.get('per_alert', {})['link']
+ lc_url = f'{BASE_BROKER_URL}/alerts/alert/{alert_name}/lightcurve.csv'
+ alert_url = f'{BASE_BROKER_URL}/{alert_link}'
elif target:
- lc_url = path.join(base_url, target.name, 'lightcurve.csv')
+ lc_url = f'{BASE_BROKER_URL}/{target.name}/lightcurve.csv'
+ alert_url = f'{BASE_BROKER_URL}/alerts/alert/{target.name}/'
else:
return
@@ -138,7 +151,7 @@ def process_reduced_data(self, target, alert=None):
'filter': 'G'
}
- rd, created = ReducedDatum.objects.get_or_create(
+ rd, _ = ReducedDatum.objects.get_or_create(
timestamp=jd.to_datetime(timezone=TimezoneInfo()),
value=json.dumps(value),
source_name=self.name,
diff --git a/tom_alerts/brokers/lasair.py b/tom_alerts/brokers/lasair.py
index ee21f8e54..c68d34d79 100644
--- a/tom_alerts/brokers/lasair.py
+++ b/tom_alerts/brokers/lasair.py
@@ -11,6 +11,15 @@ class LasairBrokerForm(GenericQueryForm):
cone = forms.CharField(required=False, label='Object Cone Search', help_text='Object RA and Dec')
sqlquery = forms.CharField(required=False, label='Freeform SQL query', help_text='SQL query')
+ def clean(self):
+ cleaned_data = super().clean()
+
+ # Ensure that either cone search or sqlquery are populated
+ if not (cleaned_data['cone'] or cleaned_data['sqlquery']):
+ raise forms.ValidationError('One of either Object Cone Search or Freeform SQL Query must be populated.')
+
+ return cleaned_data
+
def get_lasair_object(objectId):
url = LASAIR_URL + '/object/' + objectId + '/json/'
diff --git a/tom_alerts/brokers/mars.py b/tom_alerts/brokers/mars.py
index 4cbe5ccc3..2ceda4720 100644
--- a/tom_alerts/brokers/mars.py
+++ b/tom_alerts/brokers/mars.py
@@ -230,7 +230,7 @@ def process_reduced_data(self, target, alert=None):
'magnitude': candidate['candidate']['magpsf'],
'filter': filters[candidate['candidate']['fid']]
}
- rd, created = ReducedDatum.objects.get_or_create(
+ rd, _ = ReducedDatum.objects.get_or_create(
timestamp=jd.to_datetime(timezone=TimezoneInfo()),
value=json.dumps(value),
source_name=self.name,
diff --git a/tom_alerts/brokers/scimma.py b/tom_alerts/brokers/scimma.py
new file mode 100644
index 000000000..01dfaab43
--- /dev/null
+++ b/tom_alerts/brokers/scimma.py
@@ -0,0 +1,32 @@
+from crispy_forms.layout import Layout, HTML
+
+from tom_alerts.alerts import GenericBroker, GenericQueryForm, GenericAlert
+
+
+class SCIMMAQueryForm(GenericQueryForm):
+ def __init__(self, *args, **kwargs):
+ super().__init__(*args, **kwargs)
+ self.helper.inputs.pop()
+ self.helper.layout = Layout(
+ HTML('''
+
+ This plugin is a stub for the SCIMMA plugin. In order to install the full plugin, please see the
+ instructions here.
+
+ '''),
+ HTML('''Back''')
+ )
+
+
+class SCIMMABroker(GenericBroker):
+ name = 'SCIMMA'
+ form = SCIMMAQueryForm
+
+ def fetch_alerts(self, parameters):
+ return iter([])
+
+ def process_reduced_data(self, target, alert=None):
+ return
+
+ def to_generic_alert(self, alert):
+ return GenericAlert()
diff --git a/tom_alerts/brokers/tns.py b/tom_alerts/brokers/tns.py
index ab0b785c3..24c6fc724 100644
--- a/tom_alerts/brokers/tns.py
+++ b/tom_alerts/brokers/tns.py
@@ -91,7 +91,7 @@ def fetch_alerts(cls, parameters):
else:
public_timestamp = ''
data = {
- 'api_key': settings.BROKER_CREDENTIALS['TNS_APIKEY'],
+ 'api_key': settings.BROKERS['TNS']['api_key'],
'data': json.dumps({
'name': parameters['target_name'],
'internal_name': parameters['internal_name'],
@@ -108,7 +108,7 @@ def fetch_alerts(cls, parameters):
alerts = []
for transient in transients['data']['reply']:
data = {
- 'api_key': settings.BROKER_CREDENTIALS['TNS_APIKEY'],
+ 'api_key': settings.BROKERS['TNS']['api_key'],
'data': json.dumps({
'objname': transient['objname'],
'photometry': 1,
@@ -131,15 +131,16 @@ def fetch_alerts(cls, parameters):
alerts.append(alert)
else:
alerts.append(alert)
+
return iter(alerts)
@classmethod
def to_generic_alert(cls, alert):
return GenericAlert(
timestamp=alert['discoverydate'],
- url='https://wis-tns.weizmann.ac.il/object/' + alert['name'],
- id=alert['name'],
- name=alert['name_prefix'] + alert['name'],
+ url='https://wis-tns.weizmann.ac.il/object/' + alert['objname'],
+ id=alert['objname'],
+ name=alert['name_prefix'] + alert['objname'],
ra=alert['radeg'],
dec=alert['decdeg'],
mag=alert['discoverymag'],
diff --git a/tom_alerts/exceptions.py b/tom_alerts/exceptions.py
new file mode 100644
index 000000000..5f4e83087
--- /dev/null
+++ b/tom_alerts/exceptions.py
@@ -0,0 +1,4 @@
+class AlertSubmissionException(Exception):
+ """
+ The AlertSubmissionException should be used when an alert fails to be submitted to an upstream broker.
+ """
diff --git a/tom_alerts/management/commands/runbrokerquery.py b/tom_alerts/management/commands/runbrokerquery.py
index 9d81f2cf4..dda07ac1c 100644
--- a/tom_alerts/management/commands/runbrokerquery.py
+++ b/tom_alerts/management/commands/runbrokerquery.py
@@ -6,7 +6,7 @@
class Command(BaseCommand):
- help = 'Run saved alert queries and save the results as Targets'
+ help = 'Runs saved alert queries and saves the results as Targets'
def add_arguments(self, parser):
parser.add_argument(
diff --git a/tom_alerts/templates/tom_alerts/partials/submit_upstream_form.html b/tom_alerts/templates/tom_alerts/partials/submit_upstream_form.html
new file mode 100644
index 000000000..f0101d994
--- /dev/null
+++ b/tom_alerts/templates/tom_alerts/partials/submit_upstream_form.html
@@ -0,0 +1,2 @@
+{% load crispy_forms_tags %}
+{% crispy submit_upstream_form %}
\ No newline at end of file
diff --git a/tom_alerts/templatetags/alerts_extras.py b/tom_alerts/templatetags/alerts_extras.py
new file mode 100644
index 000000000..b93e94c8e
--- /dev/null
+++ b/tom_alerts/templatetags/alerts_extras.py
@@ -0,0 +1,36 @@
+from django import template
+
+from tom_alerts.alerts import get_service_class
+
+register = template.Library()
+
+
+@register.inclusion_tag('tom_alerts/partials/submit_upstream_form.html')
+def submit_upstream_form(broker, target=None, observation_record=None, redirect_url=None):
+ """
+ Renders a form to submit an alert upstream to a broker.
+ At least one of target/obs record should be given.
+
+ :param broker: The name of the broker to which the button will lead, as in the name field of the broker module.
+ :type broker: str
+
+ :param target: The target to be submitted as an alert, if any.
+ :type target: ``Target``
+
+ :param observation_record: The observation record to be submitted as an alert, if any.
+ :type observation_record: ``ObservationRecord``
+
+ :param redirect_url:
+ :type redirect_url: str
+ """
+ broker_class = get_service_class(broker)
+ form_class = broker_class.alert_submission_form
+ form = form_class(broker=broker, initial={
+ 'target': target,
+ 'observation_record': observation_record,
+ 'redirect_url': redirect_url
+ })
+
+ return {
+ 'submit_upstream_form': form
+ }
diff --git a/tom_publications/__init__.py b/tom_alerts/tests/brokers/__init__.py
similarity index 100%
rename from tom_publications/__init__.py
rename to tom_alerts/tests/brokers/__init__.py
diff --git a/tom_alerts/tests/brokers/test_alerce.py b/tom_alerts/tests/brokers/test_alerce.py
new file mode 100644
index 000000000..fd61eaac1
--- /dev/null
+++ b/tom_alerts/tests/brokers/test_alerce.py
@@ -0,0 +1,319 @@
+from datetime import datetime, timezone
+import json
+from requests import Response
+from unittest.mock import patch
+
+from astropy.time import Time
+from django.test import tag, TestCase
+from faker import Faker
+
+from tom_alerts.brokers.alerce import ALeRCEBroker, ALeRCEQueryForm
+from tom_targets.models import Target
+
+
+def create_alerce_alert(lastmjd=None, mean_magpsf_g=None, mean_magpsf_r=None, pclassrf=None, pclassearly=None):
+ fake = Faker()
+
+ return {'oid': fake.pystr_format(string_format='ZTF##???????', letters='abcdefghijklmnopqrstuvwxyz'),
+ 'meanra': fake.pyfloat(min_value=0, max_value=360),
+ 'meandec': fake.pyfloat(min_value=0, max_value=360),
+ 'lastmjd': lastmjd if lastmjd else fake.pyfloat(min_value=56000, max_value=59000, right_digits=1),
+ 'mean_magpsf_g': mean_magpsf_g if mean_magpsf_g else fake.pyfloat(min_value=16, max_value=25),
+ 'mean_magpsf_r': mean_magpsf_r if mean_magpsf_r else fake.pyfloat(min_value=16, max_value=25),
+ 'pclassrf': pclassrf if pclassrf else fake.pyfloat(min_value=0, max_value=1),
+ 'pclassearly': pclassearly if pclassearly else fake.pyfloat(min_value=0, max_value=1)}
+
+
+def create_alerce_query_response(num_alerts):
+ alerts = [create_alerce_alert() for i in range(0, num_alerts)]
+
+ return {
+ 'total': num_alerts, 'num_pages': 1, 'page': 1,
+ 'result': {alert['oid']: alert for alert in alerts}
+ }
+
+
+class TestALeRCEBrokerForm(TestCase):
+ def setUp(self):
+ self.base_form_data = {
+ 'query_name': 'Test Query',
+ 'broker': 'ALeRCE'
+ }
+
+ def test_cone_search_validation(self):
+ """Test cross-field validation for cone search filters."""
+
+ # Test that validation fails if not all fields are present
+ parameters_list = [
+ {'ra': 10, 'dec': 10}, {'dec': 10, 'sr': 10}, {'ra': 10, 'sr': 10}
+ ]
+ for parameters in parameters_list:
+ with self.subTest():
+ parameters.update(self.base_form_data)
+ form = ALeRCEQueryForm(parameters)
+ self.assertFalse(form.is_valid())
+ self.assertIn('All of RA, Dec, and Search Radius must be included to execute a cone search.',
+ form.non_field_errors())
+
+ # Test that validation passes when all three fields are present
+ self.base_form_data.update({'ra': 10, 'dec': 10, 'sr': 10})
+ form = ALeRCEQueryForm(self.base_form_data)
+ self.assertTrue(form.is_valid())
+
+ def test_time_filters_validation(self):
+ """Test validation for time filters."""
+
+ # Test that validation fails when either absolute time filter is paired with relative time filter
+ parameters_list = [
+ {'mjd__lt': 58000, 'relative_mjd__gt': 168},
+ {'mjd__gt': 58000, 'relative_mjd__gt': 168}
+ ]
+ for parameters in parameters_list:
+ with self.subTest():
+ parameters.update(self.base_form_data)
+ form = ALeRCEQueryForm(parameters)
+ self.assertFalse(form.is_valid())
+ self.assertIn('Cannot filter by both relative and absolute time.', form.non_field_errors())
+
+ # Test that mjd__lt and mjd__gt fail when mjd__lt is less than mjd__gt
+ with self.subTest():
+ parameters = {'mjd__lt': 57000, 'mjd__gt': 57001}
+ parameters.update(self.base_form_data)
+ form = ALeRCEQueryForm(parameters)
+ self.assertFalse(form.is_valid())
+ self.assertIn('Min date of first detection must be earlier than max date of first detection.',
+ form.non_field_errors())
+
+ # Test that form validation succeeds when relative time fields make sense and absolute time field is used alone.
+ parameters_list = [
+ {'mjd__gt': 57000, 'mjd__lt': 58000},
+ {'relative_mjd__gt': 168}
+ ]
+ for parameters in parameters_list:
+ with self.subTest():
+ parameters.update(self.base_form_data)
+ form = ALeRCEQueryForm(parameters)
+ self.assertTrue(form.is_valid())
+
+ # Test that form validation succeeds when absolute time field is used on its own.
+ with self.subTest():
+ parameters = {'relative_mjd__gt': 168}
+ parameters.update(self.base_form_data)
+ form = ALeRCEQueryForm(parameters)
+ self.assertTrue(form.is_valid())
+ # Test that clean_relative_mjd__gt works as expected
+ expected_mjd = Time(datetime.now()).mjd - parameters['relative_mjd__gt']/24
+ self.assertAlmostEqual(form.cleaned_data['relative_mjd__gt'], expected_mjd)
+
+
+class TestALeRCEBrokerClass(TestCase):
+ def setUp(self):
+ self.base_form_data = {
+ 'query_name': 'Test ALeRCE',
+ 'broker': 'ALeRCE',
+ }
+ self.broker = ALeRCEBroker()
+
+ def test_clean_coordinate_parameters(self):
+ """Test that _clean_date_parameters results in the correct dict structure."""
+ parameters_list = [
+ ({'ra': 10, 'dec': 10, 'sr': None}, None),
+ ({'ra': 10, 'dec': 10, 'sr': 10}, {'ra': 10, 'dec': 10, 'sr': 10})
+ ]
+ for parameters, expected in parameters_list:
+ with self.subTest():
+ self.assertEqual(self.broker._clean_coordinate_parameters(parameters), expected)
+
+ def test_clean_date_parameters(self):
+ """Test that _clean_date_parameters results in the correct dict structure."""
+ parameters_list = [
+ ({'mjd__gt': 57000, 'mjd__lt': 58000, 'relative_mjd__gt': None},
+ {'firstmjd': {'min': 57000, 'max': 58000}}),
+ ({'mjd__gt': 57000, 'mjd__lt': None, 'relative_mjd__gt': None}, {'firstmjd': {'min': 57000}}),
+ ({'mjd__gt': None, 'mjd__lt': None, 'relative_mjd__gt': 57000}, {'firstmjd': {'min': 57000}})
+ ]
+ for parameters, expected in parameters_list:
+ with self.subTest():
+ self.assertDictEqual(self.broker._clean_date_parameters(parameters), expected)
+
+ def test_clean_filter_parameters(self):
+ """Test that _clean_filter_parameters results in the correct dict structure."""
+ # Test that number of observations is populated correctly
+ parameters_list = [
+ ({'nobs__gt': 1, 'nobs__lt': 10}, {'nobs': {'min': 1, 'max': 10}}),
+ ({'nobs__gt': 1, 'nobs__lt': None}, {'nobs': {'min': 1}}),
+ ({'nobs__gt': None, 'nobs__lt': 10}, {'nobs': {'max': 10}})
+ ]
+ for parameters, expected in parameters_list:
+ with self.subTest():
+ parameters.update({k: None for k in ['classrf', 'pclassrf', 'classearly', 'pclassearly']})
+ self.assertDictContainsSubset(expected, self.broker._clean_filter_parameters(parameters))
+
+ # Test that classifiers are populated correctly
+ parameters_list = [
+ ({'classrf': 19, 'pclassrf': 0.7, 'classearly': 10, 'pclassearly': 0.5}),
+ ({'classrf': 19, 'pclassrf': 0.7, 'classearly': None, 'pclassearly': None}),
+ ({'classrf': None, 'pclassrf': None, 'classearly': 10, 'pclassearly': 0.5})
+ ]
+ for parameters in parameters_list:
+ with self.subTest():
+ parameters.update({k: None for k in ['nobs__gt', 'nobs__lt']})
+ filters = self.broker._clean_filter_parameters(parameters)
+ for key, value in parameters.items():
+ if value is not None:
+ self.assertIn(key, filters)
+ else:
+ self.assertNotIn(key, filters)
+
+ @patch('tom_alerts.brokers.alerce.ALeRCEBroker._clean_filter_parameters')
+ @patch('tom_alerts.brokers.alerce.ALeRCEBroker._clean_date_parameters')
+ @patch('tom_alerts.brokers.alerce.ALeRCEBroker._clean_coordinate_parameters')
+ def test_clean_parameters(self, mock_coordinate, mock_date, mock_filter):
+ mock_coordinate.return_value = {'ra': 10, 'dec': 10, 'sr': 10}
+ mock_date.return_value = {'firstmjd': {'min': 58000}}
+ mock_filter.return_value = {'nobs__gt': 1}
+
+ # Ensure that passed in values are used to populate the payload
+ parameters = {'page': 2, 'records': 25, 'sort_by': 'lastmjd', 'total': 30}
+ payload = self.broker._clean_parameters(parameters)
+ with self.subTest():
+ self.assertEqual(payload['page'], parameters['page'])
+ self.assertEqual(payload['records_per_pages'], parameters['records'])
+ self.assertEqual(payload['sortBy'], parameters['sort_by'])
+ self.assertEqual(payload['total'], parameters['total'])
+ self.assertIn('firstmjd', payload['query_parameters']['dates'])
+ self.assertIn('nobs__gt', payload['query_parameters']['filters'])
+ self.assertIn('coordinates', payload['query_parameters'])
+
+ # Ensure that missing values result in default values being used to populate the payload
+ mock_coordinate.return_value = None
+ payload = self.broker._clean_parameters({})
+ with self.subTest():
+ self.assertEqual(payload['page'], 1)
+ self.assertEqual(payload['records_per_pages'], 20)
+ self.assertEqual(payload['sortBy'], 'nobs')
+ self.assertNotIn('total', payload)
+ self.assertNotIn('coordinates', payload['query_parameters'])
+
+ @patch('tom_alerts.brokers.alerce.requests.post')
+ @patch('tom_alerts.brokers.alerce.ALeRCEBroker._clean_parameters')
+ def test_fetch_alerts(self, mock_clean_parameters, mock_requests_post):
+ """Test fetch_alerts broker method."""
+ mock_response = Response()
+ mock_response_content = create_alerce_query_response(25)
+ mock_response._content = str.encode(json.dumps(mock_response_content))
+ mock_response.status_code = 200
+ mock_requests_post.return_value = mock_response
+
+ response = self.broker.fetch_alerts({})
+ alerts = []
+ for alert in response:
+ alerts.append(alert)
+ self.assertDictEqual(alert, mock_response_content['result'][alert['oid']])
+ self.assertEqual(25, len(alerts))
+
+ @patch('tom_alerts.brokers.alerce.requests.post')
+ def test_fetch_alert(self, mock_requests_post):
+ """Test fetch_alert broker method."""
+ mock_response = Response()
+ mock_response_content = create_alerce_query_response(1)
+ mock_response._content = str.encode(json.dumps(mock_response_content))
+ mock_response.status_code = 200
+ mock_requests_post.return_value = mock_response
+
+ alert = self.broker.fetch_alert(list(mock_response_content['result'])[0])
+ self.assertDictEqual(list(mock_response_content['result'].items())[0][1], alert)
+
+ def test_to_target(self):
+ """Test to_target broker method."""
+ mock_alert = create_alerce_alert()
+ self.broker.to_target(mock_alert)
+ t = Target.objects.first()
+
+ self.assertEqual(mock_alert['oid'], t.name)
+ self.assertEqual(mock_alert['meanra'], t.ra)
+ self.assertEqual(mock_alert['meandec'], t.dec)
+
+ def test_to_generic_alert(self):
+ """Test to_generic_alert broker method."""
+
+ # Test that timestamp is populated correctly.
+ mock_alert = create_alerce_alert()
+ mock_alert['lastmjd'] = None
+ self.assertEqual('', self.broker.to_generic_alert(mock_alert).timestamp)
+
+ mock_alert = create_alerce_alert(lastmjd=59155)
+ self.assertEqual(datetime(2020, 11, 2, tzinfo=timezone.utc),
+ self.broker.to_generic_alert(mock_alert).timestamp)
+
+ # Test that the url is created properly.
+ mock_alert = create_alerce_alert()
+ self.assertEqual(f'https://alerce.online/object/{mock_alert["oid"]}',
+ self.broker.to_generic_alert(mock_alert).url)
+
+ # Test that the magnitude is selected correctly
+ mock_alert = create_alerce_alert(mean_magpsf_g=20, mean_magpsf_r=18)
+ self.assertEqual(mock_alert['mean_magpsf_r'], self.broker.to_generic_alert(mock_alert).mag)
+
+ mock_alert = create_alerce_alert(mean_magpsf_g=18, mean_magpsf_r=20)
+ self.assertEqual(mock_alert['mean_magpsf_g'], self.broker.to_generic_alert(mock_alert).mag)
+
+ mock_alert = create_alerce_alert(mean_magpsf_r=18)
+ mock_alert['mean_magpsf_g'] = None
+ self.assertEqual(mock_alert['mean_magpsf_r'], self.broker.to_generic_alert(mock_alert).mag)
+
+ mock_alert = create_alerce_alert(mean_magpsf_g=18, mean_magpsf_r=20)
+ mock_alert['mean_magpsf_r'] = None
+ self.assertEqual(mock_alert['mean_magpsf_g'], self.broker.to_generic_alert(mock_alert).mag)
+
+ # Test that the classification is selected correctly
+ mock_alert = create_alerce_alert()
+ self.assertEqual(mock_alert['pclassrf'], self.broker.to_generic_alert(mock_alert).score)
+
+ mock_alert = create_alerce_alert()
+ mock_alert['pclassrf'] = None
+ self.assertEqual(mock_alert['pclassearly'], self.broker.to_generic_alert(mock_alert).score)
+
+ mock_alert = create_alerce_alert()
+ mock_alert['pclassrf'] = None
+ mock_alert['pclassearly'] = None
+ self.assertEqual(None, self.broker.to_generic_alert(mock_alert).score)
+
+
+@tag('canary')
+class TestALeRCEModuleCanary(TestCase):
+ def setUp(self):
+ self.broker = ALeRCEBroker()
+
+ @patch('tom_alerts.brokers.alerce.cache.get')
+ def test_get_classifiers(self, mock_cache_get):
+ mock_cache_get.return_value = None # Ensure cache is not used
+
+ classifiers = ALeRCEQueryForm._get_classifiers()
+ self.assertIn('early', classifiers.keys())
+ self.assertIn('late', classifiers.keys())
+ for classifier in classifiers['early'] + classifiers['late']:
+ self.assertIn('name', classifier.keys())
+ self.assertIn('id', classifier.keys())
+
+ def test_fetch_alerts(self):
+ form = ALeRCEQueryForm({'query_name': 'Test', 'broker': 'ALeRCE', 'nobs__gt': 1, 'classearly': 19,
+ 'pclassearly': 0.7, 'mjd__gt': 59148.78219219812})
+ form.is_valid()
+ query = form.save()
+
+ alerts = [alert for alert in self.broker.fetch_alerts(query.parameters_as_dict)]
+
+ self.assertGreaterEqual(len(alerts), 6)
+ for k in ['oid', 'lastmjd', 'mean_magpsf_r', 'mean_magpsf_g', 'pclassrf', 'pclassearly', 'meanra', 'meandec']:
+ self.assertIn(k, alerts[0])
+
+ def test_fetch_alert(self):
+ alert = self.broker.fetch_alert('ZTF20acnsdjd')
+
+ self.assertDictContainsSubset({
+ 'oid': 'ZTF20acnsdjd',
+ 'first_magpsf_g': 17.3446006774902,
+ 'first_magpsf_r': 17.0198993682861,
+ 'firstmjd': 59149.1119328998,
+ }, alert)
diff --git a/tom_alerts/tests/brokers/test_gaia.py b/tom_alerts/tests/brokers/test_gaia.py
new file mode 100644
index 000000000..8d8e36c43
--- /dev/null
+++ b/tom_alerts/tests/brokers/test_gaia.py
@@ -0,0 +1,160 @@
+from requests import Response
+
+from django.utils import timezone
+from django.test import TestCase, override_settings
+from django.forms import ValidationError
+from unittest import mock
+
+from tom_alerts.brokers.gaia import GaiaQueryForm
+from tom_alerts.brokers.gaia import GaiaBroker
+from tom_targets.models import Target
+from tom_dataproducts.models import ReducedDatum
+
+
+@override_settings(TOM_ALERT_CLASSES=['tom_alerts.brokers.gaia.GaiaBroker'])
+class TestGaiaQueryForm(TestCase):
+ def setUp(self):
+ self.base_form_params = {'query_name': 'Test Query', 'broker': 'Gaia'}
+
+ def test_with_required_params(self):
+ self.base_form_params['target_name'] = 'Test Target'
+ form = GaiaQueryForm(self.base_form_params)
+ self.assertTrue(form.is_valid())
+
+ def test_no_query_name(self):
+ self.base_form_params['target_name'] = 'Test Target'
+ self.base_form_params.pop('query_name')
+ form = GaiaQueryForm(self.base_form_params)
+ self.assertFalse(form.is_valid())
+ self.assertIn('This field is required.', form.errors.get('query_name'))
+
+ def test_no_target_name_or_cone(self):
+ form = GaiaQueryForm(self.base_form_params)
+ self.assertFalse(form.is_valid())
+ with self.assertRaises(ValidationError):
+ form.clean()
+ self.assertIn('Please enter either a target name or cone search parameters.', form.errors.get('__all__'))
+
+ def test_both_target_name_and_cone(self):
+ self.base_form_params['target_name'] = 'Test Target'
+ self.base_form_params['cone'] = '10,20,3'
+ form = GaiaQueryForm(self.base_form_params)
+ self.assertFalse(form.is_valid())
+ with self.assertRaises(ValidationError):
+ form.clean()
+ self.assertIn('Please only enter one of target name or cone search parameters.', form.errors.get('__all__'))
+
+ def test_cone(self):
+ self.base_form_params['cone'] = '10,20,3'
+ form = GaiaQueryForm(self.base_form_params)
+ self.assertTrue(form.is_valid())
+
+ def test_cone_invalid_format(self):
+ self.base_form_params['cone'] = '10'
+ form = GaiaQueryForm(self.base_form_params)
+ self.assertFalse(form.is_valid())
+ with self.assertRaises(ValidationError):
+ form.clean()
+ self.assertIn('Cone search parameters must be in the format \'RA,Dec,Radius\'.', form.errors.get('cone'))
+
+
+@override_settings(TOM_ALERT_CLASSES=['tom_alerts.brokers.gaia.GaiaBroker'])
+class TestGaiaBroker(TestCase):
+ def setUp(self):
+ self.test_html = """
+
+
+ """
+ self.test_html = self.test_html.replace('\n', '')
+ self.alert_list = [
+ {
+ "name": "Gaia20cpu", "tnsid": "AT2020lto", "obstime": "2020-06-04 17:25:08",
+ "ra": "291.61247", "dec": "13.36801", "alertMag": "20.54", "historicMag": "17.79",
+ "historicStdDev": "0.24", "classification": "CV", "published": "2020-06-06 12:26:16",
+ "comment": "test comment", "per_alert": {
+ "link": "/alerts/alert/Gaia20cpu/", "name": "Gaia20cpu"
+ }, "rvs": 'false'},
+ {
+ "name": "Gaia20bph", "tnsid": "AT2020ftt", "obstime": "2020-04-01 12:52:23",
+ "ra": "34.02266", "dec": "68.65102", "alertMag": "16.21", "historicMag": "18.39",
+ "historicStdDev": "1.22", "classification": "unknown",
+ "published": "2020-04-03 09:47:04",
+ "comment": "candidate CV; several previous outbursts in lightcurve", "per_alert": {
+ "link": "/alerts/alert/Gaia20bph/", "name": "Gaia20bph"},
+ "rvs": 'false'}
+ ]
+ self.test_target = Target.objects.create(name=self.alert_list[0]['name'])
+ ReducedDatum.objects.create(
+ source_name='Gaia',
+ source_location=111111,
+ target=self.test_target,
+ data_type='photometry',
+ timestamp=timezone.now(),
+ value=12345.6789
+ )
+
+ @mock.patch('tom_alerts.brokers.gaia.requests.get')
+ def test_fetch_alerts(self, mock_requests_get):
+ mock_response = Response()
+ mock_response._content = self.test_html
+ mock_response.status_code = 200
+ mock_requests_get.return_value = mock_response
+
+ search_params = {'target_name': 'Gaia20bph', 'cone': None, }
+ alerts = GaiaBroker().fetch_alerts(search_params)
+ self.assertEqual(1, sum(1 for _ in alerts))
+
+ search_params = {'target_name': None, 'cone': '291.61247, 13.36801, 0.002'}
+ alerts = GaiaBroker().fetch_alerts(search_params)
+ self.assertEqual(1, sum(1 for _ in alerts))
+
+ def test_to_generic_alert(self):
+ alert = GaiaBroker().to_generic_alert(self.alert_list[0])
+ self.assertEqual(alert.name, self.alert_list[0]['name'])
+
+ @mock.patch('tom_alerts.brokers.gaia.requests.get')
+ def test_process_reduced_data_with_alert(self, mock_requests_get):
+
+ mock_photometry_response = Response()
+ mock_photometry_response._content = str.encode('''Gaia20bph\n#Date,JD,averagemag.\n
+ 2014-08-01 00:05:24,2456870.504,19.48\n2014-08-01 06:05:38,2456870.754,19.48\n\n''')
+ mock_photometry_response.status_code = 200
+ mock_requests_get.return_value = mock_photometry_response
+
+ GaiaBroker().process_reduced_data(self.test_target, alert=self.alert_list[0])
+
+ reduced_data = ReducedDatum.objects.filter(target=self.test_target, source_name='Gaia')
+ self.assertGreater(reduced_data.count(), 1)
+
+ @mock.patch('tom_alerts.brokers.gaia.requests.get')
+ @mock.patch('tom_alerts.brokers.gaia.GaiaBroker.fetch_alerts')
+ def test_process_reduced_data_without_alert(self, mock_fetch_alerts, mock_requests_get):
+ mock_fetch_alerts.return_value = iter([self.alert_list[1]])
+
+ mock_photometry_response = Response()
+ mock_photometry_response._content = str.encode('''Gaia20bph\n#Date,JD,averagemag.\n
+ 2014-08-01 00:05:24,2456870.504,19.48\n2014-08-01 06:05:38,2456870.754,19.48\n\n''')
+ mock_photometry_response.status_code = 200
+ mock_requests_get.return_value = mock_photometry_response
+
+ GaiaBroker().process_reduced_data(self.test_target)
+
+ reduced_data = ReducedDatum.objects.filter(target=self.test_target, source_name='Gaia')
+ self.assertGreater(reduced_data.count(), 1)
diff --git a/tom_alerts/tests/brokers/test_lasair.py b/tom_alerts/tests/brokers/test_lasair.py
new file mode 100644
index 000000000..fd01e6032
--- /dev/null
+++ b/tom_alerts/tests/brokers/test_lasair.py
@@ -0,0 +1,62 @@
+from unittest import mock
+
+from django.test import override_settings, tag, TestCase
+
+from tom_alerts.alerts import get_service_class
+from tom_alerts.brokers.lasair import LasairBroker, LasairBrokerForm
+
+
+class TestLasairBrokerForm(TestCase):
+ def setUp(self):
+ pass
+
+ def test_clean(self):
+ form_parameters = {'query_name': 'Test Lasair', 'broker': 'Lasair', 'name': 'ZTF18abbkloa',
+ 'cone': '', 'sqlquery': ''}
+
+ with self.subTest():
+ form = LasairBrokerForm(form_parameters)
+ self.assertFalse(form.is_valid())
+ self.assertIn('One of either Object Cone Search or Freeform SQL Query must be populated.',
+ form.non_field_errors())
+
+ test_parameters_list = [{'cone': '1, 2', 'sqlquery': ''}, {'cone': '', 'sqlquery': 'select * from objects;'}]
+ for test_params in test_parameters_list:
+ with self.subTest():
+ form_parameters.update(test_params)
+ form = LasairBrokerForm(form_parameters)
+ self.assertTrue(form.is_valid())
+
+
+@override_settings(TOM_ALERT_CLASSES=['tom_alerts.brokers.lasair.LasairBroker'])
+class TestLasairBrokerClass(TestCase):
+ """ Test the functionality of the LasairBroker, we modify the django settings to make sure
+ it is the only installed broker.
+ """
+ def setUp(self):
+ pass
+
+ def test_get_broker_class(self):
+ self.assertEqual(LasairBroker, get_service_class('Lasair'))
+
+ @mock.patch('tom_alerts.brokers.lasair.requests.get')
+ def test_fetch_alerts(self, mock_requests_get):
+ pass
+
+ def test_to_target(self):
+ pass
+
+ def test_to_generic_alert(self):
+ pass
+
+
+@tag('canary')
+class TestLasairModuleCanary(TestCase):
+ def setUp(self):
+ self.broker = LasairBroker()
+
+ def test_fetch_alerts(self):
+ pass
+
+ def test_fetch_alert(self):
+ pass
diff --git a/tom_alerts/tests/tests_mars.py b/tom_alerts/tests/brokers/test_mars.py
similarity index 54%
rename from tom_alerts/tests/tests_mars.py
rename to tom_alerts/tests/brokers/test_mars.py
index 3598ad4d6..cc612bb81 100644
--- a/tom_alerts/tests/tests_mars.py
+++ b/tom_alerts/tests/brokers/test_mars.py
@@ -1,8 +1,10 @@
+from datetime import datetime
+from itertools import islice
import json
from requests import Response
from django.utils import timezone
-from django.test import TestCase, override_settings
+from django.test import override_settings, tag, TestCase
from unittest import mock
from tom_alerts.brokers.mars import MARSBroker
@@ -115,3 +117,54 @@ def test_to_generic_alert(self):
created_alert = MARSBroker().to_generic_alert(test_alert)
self.assertEqual(created_alert.name, 'ZTF18aberpsh')
+
+
+@tag('canary')
+class TestMARSModuleCanary(TestCase):
+ def setUp(self):
+ self.broker = MARSBroker()
+ self.expected_keys = ['avro', 'candid', 'candidate', 'lco_id', 'objectId', 'publisher']
+ self.expected_candidate_keys = ['aimage', 'aimagerat', 'b', 'bimage', 'bimagerat', 'candid', 'chinr', 'chipsf',
+ 'classtar', 'clrcoeff', 'clrcounc', 'clrmed', 'clrrms', 'dec', 'decnr',
+ 'deltamaglatest', 'deltamagref', 'diffmaglim', 'distnr', 'distpsnr1',
+ 'distpsnr2', 'distpsnr3', 'drb', 'drbversion', 'dsdiff', 'dsnrms', 'elong',
+ 'exptime', 'fid', 'field', 'filter', 'fwhm', 'isdiffpos', 'jd', 'jdendhist',
+ 'jdendref', 'jdstarthist', 'jdstartref', 'l', 'magap', 'magapbig', 'magdiff',
+ 'magfromlim', 'maggaia', 'maggaiabright', 'magnr', 'magpsf', 'magzpsci',
+ 'magzpscirms', 'magzpsciunc', 'mindtoedge', 'nbad', 'ncovhist', 'ndethist',
+ 'neargaia', 'neargaiabright', 'nframesref', 'nid', 'nmatches', 'nmtchps',
+ 'nneg', 'objectidps1', 'objectidps2', 'objectidps3', 'pdiffimfilename', 'pid',
+ 'programid', 'programpi', 'ra', 'ranr', 'rb', 'rbversion', 'rcid', 'rfid',
+ 'scorr', 'seeratio', 'sgmag1', 'sgmag2', 'sgmag3', 'sgscore1', 'sgscore2',
+ 'sgscore3', 'sharpnr', 'sigmagap', 'sigmagapbig', 'sigmagnr', 'sigmapsf',
+ 'simag1', 'simag2', 'simag3', 'sky', 'srmag1', 'srmag2', 'srmag3', 'ssdistnr',
+ 'ssmagnr', 'ssnamenr', 'ssnrms', 'sumrat', 'szmag1', 'szmag2', 'szmag3',
+ 'tblid', 'tooflag', 'wall_time', 'xpos', 'ypos', 'zpclrcov', 'zpmed']
+
+ def test_fetch_alerts(self):
+ response = self.broker.fetch_alerts({'time__gt': '2018-06-01', 'time__lt': '2018-06-30'})
+
+ alerts = []
+ for alert in islice(response, 10):
+ alerts.append(alert)
+ self.assertEqual(len(alerts), 10)
+
+ for key in self.expected_keys:
+ self.assertTrue(key in alerts[0].keys())
+ for key in self.expected_candidate_keys:
+ self.assertTrue(key in alerts[0]['candidate'].keys())
+
+ def test_fetch_alert(self):
+ alert = self.broker.fetch_alert(1065519)
+
+ for key in self.expected_keys:
+ self.assertTrue(key in alert.keys())
+ for key in self.expected_candidate_keys:
+ self.assertTrue(key in alert['candidate'].keys())
+
+ def test_process_reduced_data(self):
+ alert = self.broker.fetch_alert(1065519)
+ t = Target.objects.create(name='test target', ra=1, dec=2)
+ self.broker.process_reduced_data(t, alert=alert)
+ self.assertGreaterEqual(ReducedDatum.objects.filter(target=t, timestamp__lte=datetime(2020, 11, 3)).count(),
+ 526)
diff --git a/tom_alerts/tests/tests_generic.py b/tom_alerts/tests/tests.py
similarity index 61%
rename from tom_alerts/tests/tests_generic.py
rename to tom_alerts/tests/tests.py
index 3a9ab261a..c00d0ec50 100644
--- a/tom_alerts/tests/tests_generic.py
+++ b/tom_alerts/tests/tests.py
@@ -1,12 +1,18 @@
-from django.test import TestCase, override_settings
+import json
+from unittest.mock import patch
+
from django import forms
from django.contrib.auth.models import User, Group
-from django.urls import reverse
+from django.contrib.messages import get_messages
from django.core.cache import cache
-import json
+from django.test import TestCase, override_settings
+from django.urls import reverse
-from tom_alerts.alerts import GenericQueryForm, GenericAlert, get_service_class
+from tom_alerts.alerts import GenericBroker, GenericQueryForm, GenericUpstreamSubmissionForm, GenericAlert
+from tom_alerts.alerts import get_service_class
+from tom_alerts.exceptions import AlertSubmissionException
from tom_alerts.models import BrokerQuery
+from tom_observations.models import ObservationRecord
from tom_targets.models import Target
# Test alert data. Normally this would come from a remote source.
@@ -18,20 +24,30 @@
class TestBrokerForm(GenericQueryForm):
""" All brokers must have a form which will be used to construct and save queries
- to the broker. They should sublcass `GenericQueryForm` which includes some required
+ to the broker. They should subclass `GenericQueryForm` which includes some required
fields and contains logic for serializing and persisting the query parameters to the
database. This test form will only have one field.
"""
name = forms.CharField(required=True)
-class TestBroker:
+class TestUpstreamSubmissionForm(GenericUpstreamSubmissionForm):
+ """
+ Brokers supporting upstream submission can have a form used for constructing the submission. If should subclass
+ GenericUpstreamSubmissionForm. This test form will have only one additional field in order to test that the
+ additional field value is submitted to the broker correctly.
+ """
+ topic = forms.CharField(required=False)
+
+
+class TestBroker(GenericBroker):
""" The broker class encapsulates the logic for querying remote brokers and transforming
the returned data into TOM Toolkit Targets so they can be used elsewhere in the system. The
following methods and attributes are all required, but a broker can be as complex as needed.
"""
name = 'TEST' # The name of this broker.
form = TestBrokerForm # The form that will be used to write and save queries.
+ alert_submission_form = TestUpstreamSubmissionForm
def fetch_alerts(self, parameters):
""" All brokers must implement this method. It must return a list of alerts.
@@ -58,8 +74,11 @@ def to_generic_alert(self, alert):
score=alert['score']
)
+ def submit_upstream_alert(self, target=None, observation_record=None, **kwargs):
+ return super().submit_upstream_alert(target=target, observation_record=observation_record)
+
-@override_settings(TOM_ALERT_CLASSES=['tom_alerts.tests.tests_generic.TestBroker'])
+@override_settings(TOM_ALERT_CLASSES=['tom_alerts.tests.tests.TestBroker'])
class TestBrokerClass(TestCase):
""" Test the functionality of the TestBroker, we modify the django settings to make sure
it is the only installed broker.
@@ -80,11 +99,11 @@ def test_to_generic_alert(self):
self.assertEqual(ga.name, test_alerts[0]['name'])
def test_to_target(self):
- target = TestBroker().to_generic_alert(test_alerts[0]).to_target()
+ target, _, _ = TestBroker().to_generic_alert(test_alerts[0]).to_target()
self.assertEqual(target.name, test_alerts[0]['name'])
-@override_settings(TOM_ALERT_CLASSES=['tom_alerts.tests.tests_generic.TestBroker'])
+@override_settings(TOM_ALERT_CLASSES=['tom_alerts.tests.tests.TestBroker'])
class TestBrokerViews(TestCase):
""" Test the views that use the broker classes
"""
@@ -217,3 +236,60 @@ def test_create_no_targets(self):
response = self.client.post(reverse('tom_alerts:create-target'), data=post_data, follow=True)
self.assertEqual(Target.objects.count(), 0)
self.assertRedirects(response, reverse('tom_alerts:run', kwargs={'pk': query.id}))
+
+ @patch('tom_alerts.tests.tests.TestBroker.submit_upstream_alert')
+ def test_submit_alert_success(self, mock_submit_upstream_alert):
+ """Test submission of an alert to a broker."""
+
+ # Tests that an alert is submitted with just a target, and that no redirect_url results in redirect to home
+ target = Target.objects.create(name='test_target', ra=1, dec=2)
+ response = self.client.post(reverse('tom_alerts:submit-alert', kwargs={'broker': 'TEST'}),
+ data={'target': target.id})
+
+ mock_submit_upstream_alert.assert_called_with(target=target, observation_record=None, topic='')
+ self.assertEqual(response.status_code, 302)
+ self.assertRedirects(response, reverse('home'))
+
+ # Tests that an alert is submitted with just an observation, and that redirect is to redirect_url
+ obsr = ObservationRecord.objects.create(target=target, facility='Test', parameters={}, observation_id=1)
+ response = self.client.post(reverse('tom_alerts:submit-alert', kwargs={'broker': 'TEST'}),
+ data={'observation_record': obsr.id, 'redirect_url': reverse('tom_targets:list')})
+
+ mock_submit_upstream_alert.assert_called_with(target=None, observation_record=obsr, topic='')
+ self.assertEqual(response.status_code, 302)
+ self.assertRedirects(response, reverse('tom_targets:list'))
+
+ # Tests that an alert submitted with additional parameters calls submit_upstream_alert correctly.
+ response = self.client.post(reverse('tom_alerts:submit-alert', kwargs={'broker': 'TEST'}),
+ data={'observation_record': obsr.id, 'topic': 'test topic'})
+ mock_submit_upstream_alert.assert_called_with(target=None, observation_record=obsr, topic='test topic')
+
+ @patch('tom_alerts.tests.tests.TestBroker.submit_upstream_alert')
+ def test_submit_alert_failure(self, mock_submit_upstream_alert):
+ """Test that a failed alert submission returns an appropriate message."""
+ target = Target.objects.create(name='test_target', ra=1, dec=2)
+ mock_submit_upstream_alert.return_value = False
+ response = self.client.post(reverse('tom_alerts:submit-alert', kwargs={'broker': 'TEST'}),
+ data={'target': target.id})
+
+ messages = [(m.message, m.level) for m in get_messages(response.wsgi_request)]
+ self.assertEqual(len(messages), 1)
+ self.assertEqual(messages[0][0], 'Unable to submit one or more alerts to TEST. See logs for details.')
+
+ @patch('tom_alerts.tests.tests.TestBroker.submit_upstream_alert')
+ def test_submit_alert_exception(self, mock_submit_upstream_alert):
+ """Test that an alert submission returns an appropriate message when alert submission raises an exception."""
+ mock_submit_upstream_alert.side_effect = AlertSubmissionException()
+
+ response = self.client.post(reverse('tom_alerts:submit-alert', kwargs={'broker': 'TEST'}), data={})
+ messages = [(m.message, m.level) for m in get_messages(response.wsgi_request)]
+ self.assertEqual(len(messages), 1)
+ self.assertEqual(messages[0][0], 'Unable to submit one or more alerts to TEST. See logs for details.')
+
+ def test_submit_alert_invalid_form(self):
+ """Test that an alert submission failed when form is invalid."""
+ response = self.client.post(reverse('tom_alerts:submit-alert', kwargs={'broker': 'TEST'}), data={})
+ messages = [(m.message, m.level) for m in get_messages(response.wsgi_request)]
+ self.assertEqual(len(messages), 1)
+ self.assertEqual(messages[0][0], 'Unable to submit one or more alerts to TEST. See logs for details.')
+ self.assertRedirects(response, reverse('home'))
diff --git a/tom_alerts/urls.py b/tom_alerts/urls.py
index 67c5e5baa..77c720517 100644
--- a/tom_alerts/urls.py
+++ b/tom_alerts/urls.py
@@ -1,7 +1,7 @@
from django.urls import path
-from tom_alerts.views import BrokerQueryCreateView, BrokerQueryListView, BrokerQueryUpdateView, RunQueryView
-from tom_alerts.views import CreateTargetFromAlertView, BrokerQueryDeleteView
+from tom_alerts.views import BrokerQueryCreateView, BrokerQueryListView, BrokerQueryUpdateView, BrokerQueryDeleteView
+from tom_alerts.views import CreateTargetFromAlertView, RunQueryView, SubmitAlertUpstreamView
app_name = 'tom_alerts'
@@ -11,5 +11,6 @@
path('query//update/', BrokerQueryUpdateView.as_view(), name='update'),
path('query//run/', RunQueryView.as_view(), name='run'),
path('query//delete/', BrokerQueryDeleteView.as_view(), name='delete'),
- path('alert/create/', CreateTargetFromAlertView.as_view(), name='create-target')
+ path('alert/create/', CreateTargetFromAlertView.as_view(), name='create-target'),
+ path('/submit/', SubmitAlertUpstreamView.as_view(), name='submit-alert')
]
diff --git a/tom_alerts/views.py b/tom_alerts/views.py
index 64fd0c610..0bb871a5e 100644
--- a/tom_alerts/views.py
+++ b/tom_alerts/views.py
@@ -1,6 +1,7 @@
import json
+import logging
-from django.views.generic.edit import FormView, DeleteView
+from django.views.generic.edit import DeleteView, FormMixin, FormView, ProcessFormView
from django.views.generic.base import TemplateView, View
from django.db import IntegrityError
from django.shortcuts import redirect, get_object_or_404
@@ -13,8 +14,11 @@
from django_filters.views import FilterView
from django_filters import FilterSet, ChoiceFilter, CharFilter
-from tom_alerts.models import BrokerQuery
from tom_alerts.alerts import get_service_class, get_service_classes
+from tom_alerts.models import BrokerQuery
+from tom_alerts.exceptions import AlertSubmissionException
+
+logger = logging.getLogger(__name__)
class BrokerQueryCreateView(LoginRequiredMixin, FormView):
@@ -234,9 +238,9 @@ def post(self, request, *args, **kwargs):
messages.error(request, 'Could not create targets. Try re running the query again.')
return redirect(reverse('tom_alerts:run', kwargs={'pk': query_id}))
generic_alert = broker_class().to_generic_alert(json.loads(cached_alert))
- target = generic_alert.to_target()
+ target, extras, aliases = generic_alert.to_target()
try:
- target.save()
+ target.save(extras=extras, names=aliases)
broker_class().process_reduced_data(target, json.loads(cached_alert))
for group in request.user.groups.all().exclude(name='Public'):
assign_perm('tom_targets.view_target', group, target)
@@ -255,3 +259,71 @@ def post(self, request, *args, **kwargs):
return redirect(reverse(
'tom_targets:list')
)
+
+
+class SubmitAlertUpstreamView(LoginRequiredMixin, FormMixin, ProcessFormView, View):
+ """
+ View used to submit alerts to an upstream broker, such as SCIMMA's Hopskotch or the Transient Name Server.
+
+ While this view handles the query parameters for target_id and observation_record_id by default, it will
+ send any additional query parameters to the broker, allowing a broker to use any arbitrary parameters.
+ """
+
+ def get_broker_name(self):
+ return self.kwargs['broker']
+
+ def get_form_class(self):
+ broker_name = self.get_broker_name()
+ return get_service_class(broker_name).alert_submission_form
+
+ def get_form_kwargs(self):
+ kwargs = super().get_form_kwargs()
+ kwargs.update({
+ 'broker': self.get_broker_name()
+ })
+
+ return kwargs
+
+ def get_redirect_url(self):
+ """
+ If ``next`` is provided in the query params, redirects to ``next``. If ``HTTP_REFERER`` is present on the
+ ``META`` property of the request, redirects to ``HTTP_REFERER``. Else redirects to /.
+
+ :returns: url to redirect to
+ :rtype: str
+ """
+ next_url = self.request.POST.get('redirect_url')
+ redirect_url = next_url if next_url else self.request.META.get('HTTP_REFERER')
+ if not redirect_url:
+ redirect_url = reverse('home')
+
+ return redirect_url
+
+ def form_invalid(self, form):
+ logger.log(msg=f'Form invalid: {form.errors}', level=logging.WARN)
+ messages.warning(self.request,
+ f'Unable to submit one or more alerts to {self.get_broker_name()}. See logs for details.')
+ return redirect(self.get_redirect_url())
+
+ def form_valid(self, form):
+ broker_name = self.get_broker_name()
+ broker = get_service_class(broker_name)()
+
+ target = form.cleaned_data.pop('target')
+ obsr = form.cleaned_data.pop('observation_record')
+ form.cleaned_data.pop('redirect_url') # redirect_url doesn't need to be passed to submit_upstream_alert
+
+ try:
+ # Pass non-standard fields from query parameters as kwargs
+ success = broker.submit_upstream_alert(target=target, observation_record=obsr, **form.cleaned_data)
+ except AlertSubmissionException as e:
+ logger.log(msg=f'Failed to submit alert: {e}', level=logging.WARN)
+ success = False
+
+ if success:
+ messages.success(self.request, f'Successfully submitted alerts to {broker_name}!')
+ else:
+ messages.warning(self.request,
+ f'Unable to submit one or more alerts to {self.get_broker_name()}. See logs for details.')
+
+ return redirect(self.get_redirect_url())
diff --git a/tom_base/settings.py b/tom_base/settings.py
index d6b06aed2..e8cbcc1d6 100644
--- a/tom_base/settings.py
+++ b/tom_base/settings.py
@@ -9,7 +9,7 @@
For the full list of settings and their values, see
https://docs.djangoproject.com/en/2.0/ref/settings/
"""
-
+import logging.config
import os
import tempfile
@@ -21,12 +21,12 @@
# See https://docs.djangoproject.com/en/2.0/howto/deployment/checklist/
# SECURITY WARNING: keep the secret key used in production secret!
-SECRET_KEY = 'dxja^_6p35x46dx0rx+c$(^31(10^n(twe1#ax3o8xl=n^p37q'
+SECRET_KEY = os.getenv('SECRET_KEY', 'testkey')
# SECURITY WARNING: don't run with debug turned on in production!
DEBUG = True
-ALLOWED_HOSTS = []
+ALLOWED_HOSTS = ['']
# Application definition
@@ -45,6 +45,7 @@
'django_comments',
'bootstrap4',
'crispy_forms',
+ 'rest_framework',
'django_filters',
'django_gravatar',
'tom_targets',
@@ -52,7 +53,6 @@
'tom_catalogs',
'tom_observations',
'tom_dataproducts',
- 'tom_publications',
]
SITE_ID = 1
@@ -122,6 +122,7 @@
},
]
+LOGIN_URL = '/accounts/login/'
LOGIN_REDIRECT_URL = '/'
LOGOUT_REDIRECT_URL = '/'
@@ -180,6 +181,7 @@
}
}
}
+logging.config.dictConfig(LOGGING)
TARGET_TYPE = 'SIDEREAL'
FACILITIES = {
@@ -203,19 +205,15 @@
'spectroscopy': 'tom_dataproducts.processors.spectroscopy_processor.SpectroscopyProcessor',
}
-TOM_LATEX_PROCESSORS = {
- 'ObservationGroup': 'tom_publications.processors.observation_group_latex_processor.ObservationGroupLatexProcessor',
- 'TargetList': 'tom_publications.processors.target_list_latex_processor.TargetListLatexProcessor'
-}
-
TOM_FACILITY_CLASSES = [
'tom_observations.facilities.lco.LCOFacility',
- 'tom_observations.facilities.gemini.GEMFacility'
+ 'tom_observations.facilities.gemini.GEMFacility',
+ 'tom_observations.facilities.soar.SOARFacility',
]
TOM_CADENCE_STRATEGIES = [
- 'tom_observations.cadence.RetryFailedObservationsStrategy',
- 'tom_observations.cadence.ResumeCadenceAfterFailureStrategy'
+ 'tom_observations.cadences.retry_failed_observations.RetryFailedObservationsStrategy',
+ 'tom_observations.cadences.resume_cadence_after_failure.ResumeCadenceAfterFailureStrategy'
]
# Define extra target fields here. Types can be any of "number", "string", "boolean" or "datetime"
@@ -223,7 +221,7 @@
# For example:
# EXTRA_FIELDS = [
# {'name': 'redshift', 'type': 'number', 'default': 0},
-# {'name': 'discoverer', 'type': 'string'}
+# {'name': 'discoverer', 'type': 'string'},
# {'name': 'eligible', 'type': 'boolean', 'hidden': True},
# {'name': 'dicovery_date', 'type': 'datetime'}
# ]
@@ -261,6 +259,14 @@
HINTS_ENABLED = False
HINT_LEVEL = 20
+REST_FRAMEWORK = {
+ 'DEFAULT_PERMISSION_CLASSES': [
+ ],
+ 'TEST_REQUEST_DEFAULT_FORMAT': 'json',
+ 'DEFAULT_PAGINATION_CLASS': 'rest_framework.pagination.LimitOffsetPagination',
+ 'PAGE_SIZE': 100
+}
+
try:
from local_settings import * # noqa
except ImportError:
diff --git a/tom_catalogs/admin.py b/tom_catalogs/admin.py
index 8c38f3f3d..846f6b406 100644
--- a/tom_catalogs/admin.py
+++ b/tom_catalogs/admin.py
@@ -1,3 +1 @@
-from django.contrib import admin
-
# Register your models here.
diff --git a/tom_catalogs/harvesters/tns.py b/tom_catalogs/harvesters/tns.py
index 2ec540e74..3716b90c0 100644
--- a/tom_catalogs/harvesters/tns.py
+++ b/tom_catalogs/harvesters/tns.py
@@ -1,5 +1,4 @@
import json
-import os
import requests
from astropy import units as u
@@ -8,29 +7,38 @@
from django.conf import settings
from tom_catalogs.harvester import AbstractHarvester
+from tom_common.exceptions import ImproperCredentialsException
+
+TNS_URL = 'https://wis-tns.weizmann.ac.il'
+
+try:
+ TNS_CREDENTIALS = settings.HARVESTERS['TNS']
+except (AttributeError, KeyError):
+ TNS_CREDENTIALS = {
+ 'api_key': ''
+ }
def get(term):
- api_key = settings.BROKER_CREDENTIALS['TNS_APIKEY']
- url = "https://wis-tns.weizmann.ac.il/api/get"
+ # url = "https://wis-tns.weizmann.ac.il/api/get"
+
+ get_url = TNS_URL + '/api/get/object'
- try:
- get_url = url + '/object'
+ # change term to json format
+ json_list = [("objname", term)]
+ json_file = OrderedDict(json_list)
- # change term to json format
- json_list = [("objname", term)]
- json_file = OrderedDict(json_list)
+ # construct the list of (key,value) pairs
+ get_data = [('api_key', (None, TNS_CREDENTIALS['api_key'])),
+ ('data', (None, json.dumps(json_file)))]
- # construct the list of (key,value) pairs
- get_data = [('api_key', (None, api_key)),
- ('data', (None, json.dumps(json_file)))]
+ response = requests.post(get_url, files=get_data)
+ response_data = json.loads(response.text)
- response = requests.post(get_url, files=get_data)
- response = json.loads(response.text)['data']['reply']
- return response
+ if 400 <= response_data.get('id_code') <= 403:
+ raise ImproperCredentialsException('TNS: ' + str(response_data.get('id_message')))
- except Exception as e:
- return [None, 'Error message : \n' + str(e)]
+ return response_data['data']['reply']
class TNSHarvester(AbstractHarvester):
diff --git a/tom_catalogs/models.py b/tom_catalogs/models.py
deleted file mode 100644
index 71a836239..000000000
--- a/tom_catalogs/models.py
+++ /dev/null
@@ -1,3 +0,0 @@
-from django.db import models
-
-# Create your models here.
diff --git a/tom_common/admin.py b/tom_common/admin.py
index 8c38f3f3d..e69de29bb 100644
--- a/tom_common/admin.py
+++ b/tom_common/admin.py
@@ -1,3 +0,0 @@
-from django.contrib import admin
-
-# Register your models here.
diff --git a/tom_common/exceptions.py b/tom_common/exceptions.py
index f96371ca8..c418ec8a2 100644
--- a/tom_common/exceptions.py
+++ b/tom_common/exceptions.py
@@ -1,2 +1,6 @@
class ImproperCredentialsException(Exception):
+ """
+ The ImproperCredentialsException should be used when authentication fails with an external service. This exception
+ is specifically caught by a TOM Toolkit middleware in order to render an appropriate error message.
+ """
pass
diff --git a/tom_common/middleware.py b/tom_common/middleware.py
index bb385ff70..e8ff2662d 100644
--- a/tom_common/middleware.py
+++ b/tom_common/middleware.py
@@ -18,9 +18,9 @@ def __call__(self, request):
def process_exception(self, request, exception):
if isinstance(exception, ImproperCredentialsException):
msg = (
- 'There was a problem authenticating with {}. Please check you have the correct '
- 'credentials entered into your FACILITIES setting. '
- 'https://tomtoolkit.github.io/docs/customsettings#facilities '
+ 'There was a problem authenticating with {}. Please check that you have the correct '
+ 'credentials in the corresponding settings variable. '
+ 'https://tom-toolkit.readthedocs.io/en/stable/customization/customsettings.html '
).format(
str(exception)
)
diff --git a/tom_common/models.py b/tom_common/models.py
index 71a836239..6b2021999 100644
--- a/tom_common/models.py
+++ b/tom_common/models.py
@@ -1,3 +1 @@
-from django.db import models
-
# Create your models here.
diff --git a/tom_common/serializers.py b/tom_common/serializers.py
new file mode 100644
index 000000000..fca7ad909
--- /dev/null
+++ b/tom_common/serializers.py
@@ -0,0 +1,11 @@
+from django.contrib.auth.models import Group
+from rest_framework import serializers
+
+
+class GroupSerializer(serializers.ModelSerializer):
+ id = serializers.IntegerField(required=False)
+ name = serializers.CharField(required=False)
+
+ class Meta:
+ model = Group
+ fields = ('id', 'name',)
diff --git a/tom_common/static/tom_common/css/main.css b/tom_common/static/tom_common/css/main.css
index 91cf0ea4a..b2589c6a0 100644
--- a/tom_common/static/tom_common/css/main.css
+++ b/tom_common/static/tom_common/css/main.css
@@ -1,13 +1,19 @@
body {
padding-top: 5rem;
}
+
.content {
padding: 3rem 1.5rem;
}
+
.navbar-brand > img {
- max-height: 25px;
+ max-height: 25px;
}
.input-group-text {
- font-family: monospace;
+ font-family: monospace;
+}
+
+.nav-item {
+ cursor: pointer;
}
diff --git a/tom_common/templates/comments/list.html b/tom_common/templates/comments/list.html
index aebf4e071..3af34235b 100644
--- a/tom_common/templates/comments/list.html
+++ b/tom_common/templates/comments/list.html
@@ -7,9 +7,9 @@
{{ comment.user.first_name }} {{ comment.user.last_name }}
{% if not object %}
- on {{ comment.content_object.identifier }}
+ about {{ comment.content_object.name }}
{% endif %}
- {{ comment.submit_date|date }}
+ on {{ comment.submit_date|date }}
{% if comment.user == user or user.is_superuser %}
✖
{% endif %}
diff --git a/tom_common/templates/tom_common/base.html b/tom_common/templates/tom_common/base.html
index deedcf3f3..29d871c80 100644
--- a/tom_common/templates/tom_common/base.html
+++ b/tom_common/templates/tom_common/base.html
@@ -75,8 +75,8 @@
{% block javascript %}
- {% endblock %}
- {% block extra_javascript %}
- {% endblock %}
+ {% endblock %}
+ {% block extra_javascript %}
+ {% endblock %}
| 
-
-
- 
-
-