Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[17.0][MIG] base_import_async: Migration to V17 #681

Open
wants to merge 87 commits into
base: 17.0
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
87 commits
Select commit Hold shift + click to select a range
6a53217
[IMP] rename base_import_connector to base_import_async
sbidoul Nov 14, 2014
ce9c7c6
[FIX] silence pylint, which btw discovered a genuine bug
sbidoul Nov 14, 2014
20910e6
[IMP] code style using session shortcuts
sbidoul Nov 14, 2014
de257d4
[IMP] slightly improve README
sbidoul Nov 14, 2014
6288597
[IMP] add comma in README
sbidoul Nov 14, 2014
65993ef
Add OCA as author of OCA addons
gurneyalex Mar 3, 2015
fba1dda
[IMP] OpenERP -> Odoo in comments
sbidoul Apr 20, 2015
af559b2
[FIX] base_import_async: properly install connector module
sbidoul Apr 20, 2015
eea1e8a
[IMP] base_import_async: update README
sbidoul Apr 12, 2015
c21b45b
[IMP] base_import_async: further align README with OCA template
sbidoul Apr 21, 2015
dbd8895
base_import_async: allow to set the base priority
guewen May 15, 2015
119f0ad
Add bug tracker link on README.rst
yvaucher May 22, 2015
f80d9ca
Replace deprecated functions
guewen Jun 10, 2015
9a10689
Remove call to install_in_connector()
guewen Jun 10, 2015
1e1d5a2
[UPD] prefix versions with 8.0
sbidoul Oct 9, 2015
d8e779f
[MIG] Make modules uninstallable
pedrobaeza Oct 14, 2015
015c904
[FIX] Fix new/old api compatibility
lmignon Oct 19, 2015
8a3f563
Update base_import_async.py
OdooMod Nov 19, 2015
6b23059
Update base_import_async.py
OdooMod Nov 19, 2015
7dae067
migrate base_import_async and it's test to 9.0
sbidoul Mar 3, 2016
9683ddf
migrate javascript
sbidoul Mar 3, 2016
3436610
[MIG] Make modules uninstallable
pedrobaeza Oct 6, 2016
9843a13
[MIG] Rename manifest files
pedrobaeza Oct 6, 2016
fc53c4b
Porting to new API
danieltorres7 Nov 4, 2016
9a8ed10
Whitespace
danieltorres7 Nov 7, 2016
7b6ee78
[MIG] base_import_async: replace connector by queue_job
lmignon Mar 21, 2017
3250525
[FIX] base_import_async: with_delay returns an instance of Job
lmignon Mar 23, 2017
9926b50
[MIG] base_import_async and test_base_import_async to 11.0
tarteo Jun 21, 2018
1406a8b
[IMP] Processed review
tarteo Jun 22, 2018
8e467ed
[IMP] Readability, speed, and added HISTORY.rst
tarteo Jun 26, 2018
d2f225c
[UPD] README.rst
OCA-git-bot Jun 27, 2018
38b1d52
[MIG] Make modules uninstallable
OCA-git-bot Sep 27, 2018
d585ff0
Set modules as uninstallable
guewen Oct 1, 2019
fd698ac
Run pre-commit with black, isort, ...
guewen Oct 4, 2019
6cd85a1
Migrate base_import_async to 12.0
guewen Oct 2, 2018
eec720f
Add OCA development status
guewen Oct 5, 2018
f034b2d
Exit the import screen after importing file in async
guewen Nov 15, 2019
63f2ace
Set default values for encoding and separator
guewen Nov 17, 2019
01369d1
[UPD] Update base_import_async.pot
oca-travis Nov 17, 2019
3dc2b5e
[UPD] README.rst
OCA-git-bot Nov 17, 2019
6fbf86a
[ADD] icon.png
OCA-git-bot Nov 17, 2019
4ae6554
[IMP] base_import_async: black, isort
guewen Dec 20, 2019
da8428e
[MIG] base_import_async: Migration to 13.0
guewen Dec 20, 2019
6fa3340
Apply suggestions from code review
guewen Jan 9, 2020
91f138e
[UPD] Update base_import_async.pot
oca-travis Jan 10, 2020
8155149
[UPD] README.rst
OCA-git-bot Jan 10, 2020
bb21477
pre-commit update
OCA-git-bot Mar 7, 2020
18b93db
[FIX] base_import_async error message
Jun 25, 2020
8d0bfc4
base_import_async 13.0.1.0.1
OCA-git-bot Jun 26, 2020
32af96a
base_import_async: remove `@job` decorator
guewen Nov 2, 2020
4f751f7
Fix invalid development_status (accepted value is Production/Stable)
guewen Nov 2, 2020
3e0dfcf
[UPD] README.rst
OCA-git-bot Nov 10, 2020
8a119c1
base_import_async 13.0.2.0.0
OCA-git-bot Nov 12, 2020
242ae24
[MIG] init V14 repo
sebastienbeau Nov 20, 2020
76c2542
[MIG] base_import_async: Migration to 14.0
BT-jcamacho May 27, 2021
dd3e4a4
[UPD] Update base_import_async.pot
oca-travis Aug 11, 2021
3ccc8ca
[UPD] README.rst
OCA-git-bot Aug 11, 2021
2888ba5
base_import_async 14.0.1.0.1
OCA-git-bot Aug 11, 2021
2bda20f
Set modules as uninstallable
guewen Nov 1, 2021
8ea189a
Added translation using Weblate (Italian)
francesco-ooops Dec 3, 2021
c4cc2e3
Translated using Weblate (Italian)
francesco-ooops Dec 3, 2021
a2cab4d
[IMP] update dotfiles [ci skip]
OCA-git-bot Mar 31, 2022
3dababe
base_import_async: add a dedicated channel
sebalix Jul 29, 2022
b82f651
base_import_async 14.0.1.0.2
OCA-git-bot Nov 22, 2022
6780e08
Added translation using Weblate (Romanian)
dhongu Nov 22, 2022
3dcaf96
[IMP] base_import_async: black, isort, prettier
ieski Nov 24, 2022
3bcbfe2
[MIG] base_import_async: Migration to 15.0
ieski Nov 24, 2022
89d7c71
[16.0][MIG] base_import_async: Migration to 16.0
dzungtran89 Mar 1, 2023
be2585d
[UPD] Update base_import_async.pot
Jun 28, 2023
7bc1c09
[UPD] README.rst
OCA-git-bot Jun 28, 2023
520b954
Update translation files
weblate Jun 28, 2023
694d3eb
[UPD] README.rst
OCA-git-bot Sep 3, 2023
845cab2
[FIX] base_import_async: restore class
danielduqma Sep 13, 2023
8ffb028
[FIX] base_import_async: remove default values from context when crea…
marielejeune Apr 14, 2023
8adec3b
oca-github-bot post-merge updates
OCA-git-bot Sep 18, 2023
a6135dc
[BOT] post-merge updates
OCA-git-bot Sep 25, 2023
74eb9ac
Added translation using Weblate (Spanish)
asierneiradev Sep 25, 2023
a24190d
Translated using Weblate (Spanish)
asierneiradev Sep 25, 2023
5ec3ace
Update translation files
weblate Oct 9, 2023
7a05080
Added translation using Weblate (Italian)
mymage Jan 9, 2024
e414b44
Translated using Weblate (Italian)
mymage Jan 11, 2024
a3c8702
[16.0][FIX] base_import_async: update OPT_HAS_HEADERS option
alexbottemanne Jan 31, 2024
f7ff55f
[BOT] post-merge updates
OCA-git-bot Apr 4, 2024
899309c
[16.0][FIX] base_import_async, add sudo call to models without explic…
ChrisOForgeFlow Jul 10, 2024
eef1849
[BOT] post-merge updates
OCA-git-bot Jul 10, 2024
bf4ffa1
[IMP] base_import_async: pre-commit stuff
Wodran14 Sep 3, 2024
fb27984
[MIG] base_import_async: Migration to 17.0
Wodran14 Sep 3, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
153 changes: 153 additions & 0 deletions base_import_async/README.rst
Original file line number Diff line number Diff line change
@@ -0,0 +1,153 @@
===================
Asynchronous Import
===================

..
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! This file is generated by oca-gen-addon-readme !!
!! changes will be overwritten. !!
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!! source digest: sha256:e7bcaae80fd59cc8e45b2784606eb026fba7dfcac892ad4a986f51e0dc248e1e
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

.. |badge1| image:: https://img.shields.io/badge/maturity-Production%2FStable-green.png
:target: https://odoo-community.org/page/development-status
:alt: Production/Stable
.. |badge2| image:: https://img.shields.io/badge/licence-AGPL--3-blue.png
:target: http://www.gnu.org/licenses/agpl-3.0-standalone.html
:alt: License: AGPL-3
.. |badge3| image:: https://img.shields.io/badge/github-OCA%2Fqueue-lightgray.png?logo=github
:target: https://github.com/OCA/queue/tree/17.0/base_import_async
:alt: OCA/queue
.. |badge4| image:: https://img.shields.io/badge/weblate-Translate%20me-F47D42.png
:target: https://translation.odoo-community.org/projects/queue-17-0/queue-17-0-base_import_async
:alt: Translate me on Weblate
.. |badge5| image:: https://img.shields.io/badge/runboat-Try%20me-875A7B.png
:target: https://runboat.odoo-community.org/builds?repo=OCA/queue&target_branch=17.0
:alt: Try me on Runboat

|badge1| |badge2| |badge3| |badge4| |badge5|

This module extends the standard CSV import functionality to import
files in the background using the OCA/queue framework.

**Table of contents**

.. contents::
:local:

Usage
=====

The user is presented with a new checkbox in the import screen. When
selected, the import is delayed in a background job.

This job in turn splits the CSV file in chunks of minimum 100 lines (or
more to align with record boundaries). Each chunk is then imported in a
separate background job.

When an import fails, the job is marked as such and the user can read
the error in the job status. The CSV chunk being imported is stored as
an attachment to the job, making it easy to download it, fix it and run
a new import, possibly in synchronous mode since the chunks are small.

Any file that can be imported by the standard import mechanism can also
be imported in the background.

This module's scope is limited to making standard imports asynchronous.
It does not attempt to transform the data nor automate ETL flows.

Other modules may benefit from this infrastructure in the following way
(as illustrated in the test suite):

1. create an instance of base_import.import and populate its fields
(res_model, file, file_name),
2. invoke the do method with appropriate options (header, encoding,
separator, quoting, use_queue, chunk_size).

Known issues / Roadmap
======================

- There is currently no user interface to control the chunk size, which
is currently 100 by default. Should this proves to be an issue, it is
easy to add an option to extend the import screen.
- Validation cannot be run in the background.

Changelog
=========

13.0.1.0.0 (2019-12-20)
-----------------------

- [MIGRATION] from 12.0 branched at rev. a7f8031

Bug Tracker
===========

Bugs are tracked on `GitHub Issues <https://github.com/OCA/queue/issues>`_.
In case of trouble, please check there if your issue has already been reported.
If you spotted it first, help us to smash it by providing a detailed and welcomed
`feedback <https://github.com/OCA/queue/issues/new?body=module:%20base_import_async%0Aversion:%2017.0%0A%0A**Steps%20to%20reproduce**%0A-%20...%0A%0A**Current%20behavior**%0A%0A**Expected%20behavior**>`_.

Do not contact contributors directly about support or help with technical issues.

Credits
=======

Authors
-------

* Akretion
* ACSONE SA/NV

Contributors
------------

Sébastien Beau (Akretion) authored the initial prototype.

Stéphane Bidoul (ACSONE) extended it to version 1.0 to support
multi-line records, store data to import as attachments and let the user
control the asynchronous behaviour.

Other contributors include:

- Anthony Muschang (ACSONE)

- David Béal (Akretion)

- Jonathan Nemry (ACSONE)

- Laurent Mignon (ACSONE)

- Dennis Sluijk (Onestein)

- Guewen Baconnier (Camptocamp)

- `Trobz <https://trobz.com>`__:

- Dzung Tran <[email protected]>

- Daniel Duque (FactorLibre)

Other credits
-------------

The migration of this module from 15.0 to 16.0 was financially supported
by Camptocamp

Maintainers
-----------

This module is maintained by the OCA.

.. image:: https://odoo-community.org/logo.png
:alt: Odoo Community Association
:target: https://odoo-community.org

OCA, or the Odoo Community Association, is a nonprofit organization whose
mission is to support the collaborative development of Odoo features and
promote its widespread use.

This module is part of the `OCA/queue <https://github.com/OCA/queue/tree/17.0/base_import_async>`_ project on GitHub.

You are welcome to contribute. To learn how please visit https://odoo-community.org/page/Contribute.
3 changes: 3 additions & 0 deletions base_import_async/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,3 @@
# License AGPL-3.0 or later (https://www.gnu.org/licenses/agpl).

from . import models
25 changes: 25 additions & 0 deletions base_import_async/__manifest__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,25 @@
# @author Stéphane Bidoul <[email protected]>
# @author Sébastien BEAU <[email protected]>
# License AGPL-3.0 or later (https://www.gnu.org/licenses/agpl).

{
"name": "Asynchronous Import",
"summary": "Import CSV files in the background",
"version": "17.0.1.0.0",
"author": "Akretion, ACSONE SA/NV, Odoo Community Association (OCA)",
"license": "AGPL-3",
"website": "https://github.com/OCA/queue",
"category": "Generic Modules",
"depends": ["base_import", "queue_job"],
"data": [
"data/queue_job_function_data.xml",
],
"assets": {
"web.assets_backend": [
"base_import_async/static/src/js/import_model.esm.js",
"base_import_async/static/src/xml/import_data_sidepanel.xml",
],
},
"installable": True,
"development_status": "Production/Stable",
}
21 changes: 21 additions & 0 deletions base_import_async/data/queue_job_function_data.xml
Original file line number Diff line number Diff line change
@@ -0,0 +1,21 @@
<odoo noupdate="1">
<record id="job_function_base_import_import_split_file" model="queue.job.function">
<field name="model_id" ref="base_import.model_base_import_import" />
<field name="method">_split_file</field>
<field
name="related_action"
eval='{"func_name": "_related_action_attachment"}'
/>
</record>
<record
id="job_function_base_import_import_import_one_chunk"
model="queue.job.function"
>
<field name="model_id" ref="base_import.model_base_import_import" />
<field name="method">_import_one_chunk</field>
<field
name="related_action"
eval='{"func_name": "_related_action_attachment"}'
/>
</record>
</odoo>
78 changes: 78 additions & 0 deletions base_import_async/i18n/base_import_async.pot
Original file line number Diff line number Diff line change
@@ -0,0 +1,78 @@
# Translation of Odoo Server.
# This file contains the translation of the following modules:
# * base_import_async
#
msgid ""
msgstr ""
"Project-Id-Version: Odoo Server 16.0\n"
"Report-Msgid-Bugs-To: \n"
"Last-Translator: \n"
"Language-Team: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: \n"
"Plural-Forms: \n"

#. module: base_import_async
#. odoo-python
#: code:addons/base_import_async/models/queue_job.py:0
#, python-format
msgid "Attachment"
msgstr ""

#. module: base_import_async
#: model:ir.model,name:base_import_async.model_base_import_import
msgid "Base Import"
msgstr ""

#. module: base_import_async
#. odoo-python
#: code:addons/base_import_async/models/base_import_import.py:0
#, python-format
msgid ""
"Import %(model)s from file %(file_name)s - #%(chunk)s - lines %(from)s to "
"%(to)s"
msgstr ""

#. module: base_import_async
#. odoo-python
#: code:addons/base_import_async/models/base_import_import.py:0
#, python-format
msgid "Import %(model)s from file %(from_file)s"
msgstr ""

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/xml/import.xml:0
#, python-format
msgid "Import in the background"
msgstr ""

#. module: base_import_async
#: model:ir.model,name:base_import_async.model_queue_job
msgid "Queue Job"
msgstr ""

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/xml/import.xml:0
#, python-format
msgid ""
"When checked, the import will be executed as a background job, after "
"splitting your file in small chunks that will be processed independently. "
"Use this to import very large files."
msgstr ""

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/js/import.js:0
#, python-format
msgid "You can check the status of this job in menu 'Queue / Jobs'."
msgstr ""

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/js/import.js:0
#, python-format
msgid "Your request is being processed"
msgstr ""
88 changes: 88 additions & 0 deletions base_import_async/i18n/es.po
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
# Translation of Odoo Server.
# This file contains the translation of the following modules:
# * base_import_async
#
msgid ""
msgstr ""
"Project-Id-Version: Odoo Server 16.0\n"
"Report-Msgid-Bugs-To: \n"
"PO-Revision-Date: 2023-09-25 09:34+0000\n"
"Last-Translator: Asier Neira <[email protected]>\n"
"Language-Team: none\n"
"Language: es\n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: \n"
"Plural-Forms: nplurals=2; plural=n != 1;\n"
"X-Generator: Weblate 4.17\n"

#. module: base_import_async
#. odoo-python
#: code:addons/base_import_async/models/queue_job.py:0
#, python-format
msgid "Attachment"
msgstr "Archivo adjunto"

#. module: base_import_async
#: model:ir.model,name:base_import_async.model_base_import_import
msgid "Base Import"
msgstr "Importación Base"

#. module: base_import_async
#. odoo-python
#: code:addons/base_import_async/models/base_import_import.py:0
#, python-format
msgid ""
"Import %(model)s from file %(file_name)s - #%(chunk)s - lines %(from)s to "
"%(to)s"
msgstr ""
"Importar %(model)s del archivo %(file_name)s - #%(chunk)s - líneas %(from)s "
"a %(to)s"

#. module: base_import_async
#. odoo-python
#: code:addons/base_import_async/models/base_import_import.py:0
#, python-format
msgid "Import %(model)s from file %(from_file)s"
msgstr "Importar %(model)s del fichero %(from_file)s"

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/xml/import.xml:0
#, python-format
msgid "Import in the background"
msgstr "Importación en segundo plano"

#. module: base_import_async
#: model:ir.model,name:base_import_async.model_queue_job
msgid "Queue Job"
msgstr "Cola de trabajo"

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/xml/import.xml:0
#, python-format
msgid ""
"When checked, the import will be executed as a background job, after "
"splitting your file in small chunks that will be processed independently. "
"Use this to import very large files."
msgstr ""
"Si esta opción está seleccionada, la importación se ejecutará como un "
"trabajo en segundo plano, tras dividir el archivo en pequeños trozos que se "
"procesarán de forma independiente. Utilícelo para importar archivos muy "
"grandes."

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/js/import.js:0
#, python-format
msgid "You can check the status of this job in menu 'Queue / Jobs'."
msgstr ""
"Puede comprobar el estado de este trabajo en el menú 'Cola / Trabajos'."

#. module: base_import_async
#. odoo-javascript
#: code:addons/base_import_async/static/src/js/import.js:0
#, python-format
msgid "Your request is being processed"
msgstr "Su solicitud está siendo procesada"
Loading
Loading