README for NonameTV
-------------------

LICENSE

This program is free software: you can redistribute it and/or modify
it under the terms of the GNU Affero General Public License as
published by the Free Software Foundation, either version 3 of the
License, or (at your option) any later version.

This program is distributed in the hope that it will be useful, but
WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the GNU
Affero General Public License for more details.

You should have received a copy of the GNU Affero General Public
License along with this program.  If not, see
<http://www.gnu.org/licenses/>.

INTRODUCTION

The system consists of three main parts: the Importers, the Database
and the Exporters.

IMPORTERS

Importers are used to import data from different sources into the database.
There is one importers for each data provider. All importers inherit
from the NonameTV::Importer class and thus provide a consistent interface.

Data is imported in batches. A batch is the smallest unit of data that
can be identified by the importer. Usually, a batch corresponds to
a file provided by the data provider.

All Importers that import data via HTTP inherit from one of
NonameTV::Importer::BaseOne, BaseDaily or BaseWeekly. These baseclasses
provide helper-functions that handle data provided in a single file per
channel, one file per day and channel or one file per week and channel.

There is an override-mechanism implemented in the importers that can be
used to override faulty data provided by the data-providers. Sometimes
the data is plain wrong and you need to edit it before feeding it to
the importers. See tools/nonametv-import-download for instructions.

The Importers are run via the command-line tool tools/nonametv-import:

perl -I lib tools/nonametv-import Infomedia --verbose

DATABASE

All data is stored in a MySQL-database in a consistent format. All
strings are encoded in utf-8 in the database and all times are stored
in UTC-timezone.  The channels and trans_cat tables are updated
manually (using the mysql frontend of your choice.) The programs and
batches tables are updated by the Importers and cannot be updated
manually.

EXPORTERS

The Exporters take the data in the database and export it in another format.
The only Exporter that exist today is the Xmltv exporter that exports data
in a format that tv_grab_se_swedb can utilize.

AVOIDING UNNECESSARY UPDATES

tv_grab_se_swedb utilizes http-caching to avoid downloading data that
hasn't changed since the last time tv_grab_se_swedb was run. This
saves a lot of bandwidth for the server if utilized correctly.

Additionally, some of the Importers take a lot of time to run and we want
to avoid Importing data if nothing has changed. There are a number of
mechanisms in place to avoid doing unnecessary work:

All Importers use HTTP-caching (if they fetch data via HTTP) and only
processes data if it has actually changed since it was last downloaded.
To force an update (for example if the Importer-code has been improved),
use the parameter --force-update:

perl -I lib tools/nonametv-import Infomedia --force-update

The Exporter only exports data for batches that have actually changed.
Furthermore, after exporting a file, the Exporter checks to see if the
exported file is different from the previous file for that period. If
the new file is identical to the old file, the old file is left untouched.

After the export-step, rsync is used to copy the data from the
xmltv_staging directory to the xmltv-directory. Once more, only files
that have actually changed (detected by checking file contents) are
copied. This can be useful during development, since you can do an
export, see that something went wrong (for example by using 
tools/nonametv-xmltv-compare-run and look at the changes). Then revert
your changes and run export again. Since the newly exported files are
then identical to the files in the xmltv/directory, the next run of
rsync will not copy any files even though the date on the files has been
updated.