Closed Access

Closed Access screenshot

Tim Sherratt | | University of Canberra | timsherratt.org

play along

wragge.github.io/dha2016/

You might need to let your browser load ‘insecure’ content to view everything.
It’s just a http/https thing…

a good type

NAA: A463, 1957/34 PART 1

a right to access

Canberra Times, 31 December 1970, p. 1.
http://nla.gov.au/nla.news-article110446260

the public interest

NAA: A463, 1957/34 PART 1

Archives Act 1983

33 Exempt records
    (1) For the purposes of this Act, a Commonwealth record is an exempt
        record if it contains information or matter of any of the following
        kinds:
        (a) information or matter the disclosure of which under this Act
            could reasonably be expected to cause damage to the
            security, defence or international relations of the
            Commonwealth;
        (b) information or matter:
            (i) that was communicated in confidence by, or on behalf
                of, a foreign government, an authority of a foreign
                government or an international organisation (the foreign
                entity) to the Government of the Commonwealth, to an
                authority of the Commonwealth or to a person who
                received the communication on behalf of the
                Commonwealth or an authority of the Commonwealth
                (the Commonwealth entity); and
            (ii) which the foreign entity advises the Commonwealth
                entity is still confidential; and
            (iii) the confidentiality of which it would be reasonable to
                maintain;
        (c) information or matter the disclosure of which under this Act
            would have a substantial adverse effect on the financial or
            property interests of the Commonwealth or of a
            Commonwealth institution and would not, on balance, be in
            the public interest;
        (d) information or matter the disclosure of which under this Act
            would constitute a breach of confidence;
        (e) information or matter the disclosure of which under this Act
            would, or could reasonably be expected to:
            (i) prejudice the conduct of an investigation of a breach, or
                possible breach, of the law, or a failure, or possible
                failure, to comply with a law relating to taxation or
                prejudice the enforcement or proper administration of
                the law in a particular instance;
            (ii) disclose, or enable a person to ascertain, the existence or
                identity of a confidential source of information in
                relation to the enforcement or administration of the law;
                or
            (iii) endanger the life or physical safety of any person;
        (f) information or matter the disclosure of which under this Act
            would, or could reasonably be expected to:
            (i) prejudice the fair trial of a person or the impartial
            adjudication of a particular case;
            (ii) disclose lawful methods or procedures for preventing,
                detecting, investigating, or dealing with matters arising
                out of, breaches or evasions of the law the disclosure of
                which would, or would be reasonably likely to,
                prejudice the effectiveness of those methods or
                procedures; or
            (iii) prejudice the maintenance or enforcement of lawful
                methods for the protection of public safety;
        (g) information or matter the disclosure of which under this Act
            would involve the unreasonable disclosure of information
            relating to the personal affairs of any person (including a
            deceased person);
        (h) information or matter relating to trade secrets, or any other
            information or matter having a commercial value that would
            be, or could reasonably be expected to be, destroyed or
            diminished if the information or matter were disclosed;
        (j) information or matter (other than information or matter
            referred to in paragraph (h)) concerning a person in respect of
            his or her business or professional affairs or concerning the
            business, commercial or financial affairs of an organization
            or undertaking, being information or matter the disclosure of
            which would, or could reasonably be expected to,
            unreasonably affect that person adversely in respect of his or
            her lawful business or professional affairs or that
            organization or undertaking in respect of its lawful business,
            commercial or financial affairs.
    (2) For the purposes of this Act, a Commonwealth record is an exempt
        record if it is of such a nature that:
        (a) it would be privileged from production in legal proceedings
            on the ground of legal professional privilege; and
        (b) disclosure of the record would be contrary to the public
            interest.
    (3) For the purposes of this Act, a Commonwealth record is an exempt
        record if:
        (a) it contains information or matter:
            (i) that relates to the personal affairs, or the business or
                professional affairs, of any person (including a deceased
                person); or
            (ii) that relates to the business, commercial or financial
                affairs of an organization or undertaking; and
        (b) there is in force a law relating to taxation that applies
            specifically to information or matter of that kind and
            prohibits persons referred to in that law from disclosing
            information or matter of that kind, whether the prohibition is
            absolute or is subject to exceptions or qualifications.
http://www.austlii.edu.au/au/legis/cth/consol_act/aa198398/
https://plot.ly/~wragge/421/

open access

NAA: A1209, 1969/8205 PART 1

RecordSearch access status dropdown

http://recordsearch.naa.gov.au/scripts/Logon.asp?N=guest
class SearchHarvester():
    """
    Harvest the details of 'Closed' files from RecordSearch.
    Saves to MongoDB.
    harvester = SearchHarvester(harvest='2015-12-31')
    harvester.start_harvest()
    """
    def __init__(self, harvest, **kwargs):
        self.total_pages = None
        self.pages_complete = 0
        self.client = RSSearchClient()
        self.prepare_harvest(access='Closed')
        db = self.get_db()
        self.items = db.items
        self.series = db.series
        self.harvests = db.harvests
        self.set_harvest(harvest)

    def set_harvest(self, harvest):
        # self.harvests.create_index('harvest_date', unique=True)
        dates = harvest.split('-')
        harvest_date = datetime(int(dates[0]), int(dates[1]), int(dates[2]), 0, 0, 0)
        try:
            self.harvests.insert({'harvest_date': harvest_date})
        except DuplicateKeyError:
            pass
        self.harvest_date = harvest_date

    def get_db(self):
        dbclient = MongoClient(MONGOLAB_URL)
        db = dbclient.get_default_database()
        return db

    def get_total(self):
        return self.client.total_results

    def prepare_harvest(self, **kwargs):
        self.client.search(**kwargs)
        total_results = self.client.total_results
        print '{} items'.format(total_results)
        self.total_pages = (int(total_results) / self.client.results_per_page) + 1
        print '{} pages'.format(self.total_pages)

    def start_harvest(self, page=None):
        item_client = RSItemClient()
        series_client = RSSeriesClient()
        # Refresh series with each harvest
        # self.series.remove({})
        # self.items.remove({})
        if not page:
            page = self.pages_complete + 1
        else:
            self.pages_complete = page - 1
        while self.pages_complete < self.total_pages:
            response = self.client.search(access='Closed', page=page, sort='9')
            for result in response['results']:
                item = item_client.get_summary(entity_id=result['identifier'])
                item['_id'] = item['identifier']
                item['random_id'] = [random.random(), 0]
                # Normalise reasons
                item['reasons'] = []
                # item['year'] = item['contents_dates']['end_date']['date'].year
                for reason in item['access_reason']:
                    matched = False
                    for exception, pattern in EXCEPTIONS:
                        if re.match(pattern, reason['reason']):
                            item['reasons'].append(exception)
                            matched = True
                    if not matched:
                        item['reasons'].append(reason['reason'])
                # Get series and agency info
                print item['series']
                series = self.series.find_one({'identifier': item['series']})
                if not series:
                    try:
                        series = series_client.get_summary(entity_id=item['series'])
                        agencies = series_client.get_controlling_agencies(entity_id=item['series'])
                        series['controlling_agencies'] = agencies
                        self.series.insert(series)
                    except UsageError:
                        series = None
                if series:
                    item['series_title'] = series['title']
                    if series['controlling_agencies']:
                        item['agencies'] = []
                        for agency in series['controlling_agencies']:
                            if not agency['end_date']:
                                item['agencies'].append(agency)
                old_item = self.items.find_one_and_replace({'_id': item['identifier']}, item, upsert=True)
                if old_item and ('harvests' in old_item) and (self.harvest_date not in old_item['harvests']):
                    harvests = old_item['harvests'].append(self.harvest_date)
                else:
                    harvests = [self.harvest_date]
                self.items.update_one({'_id': item['identifier']}, {'$set': {'harvests': harvests}})
                print item['identifier']
            self.pages_complete += 1
            page += 1
            print '{} pages complete'.format(self.pages_complete)
            time.sleep(1)
https://github.com/wragge/closed_access
https://dx.doi.org/10.6084/m9.figshare.3443867.v1

closed access

https://closedaccess.herokuapp.com/
http://closedaccess.herokuapp.com/items/?q=&reasons=33(1)(a)&decision_after=1989-05-01&decision_before=1989-05-31&sort=decisions
http://timsherratt.org/research-notebook/notes/interpreting-annual-report-totals/

reasons

http://closedaccess.herokuapp.com/reasons/

Visualisation of relationships between reason

http://timsherratt.org/research-notebook/notes/relationships-between-reasons/
http://closedaccess.herokuapp.com/reasons/

pre-access recorder

http://closedaccess.herokuapp.com/reasons/Pre Access Recorder/
http://timsherratt.org/research-notebook/experiments/titles-pre-access-recorder-cleaned/

withheld pending advice

https://closedaccess.herokuapp.com/reasons/Withheld pending adv/
http://timsherratt.org/research-notebook/notes/interpreting-annual-report-totals/
https://closedaccess.herokuapp.com/items/?q=&reasons=Withheld+pending+adv&reasons_match=exact&decision_before=2013-01-01&sort=decisions
https://closedaccess.herokuapp.com/series/A1838/
http://timsherratt.org/research-notebook/notes/the-conservatism-of-access-decisions/
https://closedaccess.herokuapp.com/mostconservative/
http://timsherratt.org/research-notebook/projects/immigration-and-surveillance/

no magic