{
    "info": {
        "author": "chimera0",
        "author_email": "ai-brain-lab@accel-brain.com",
        "bugtrack_url": null,
        "classifiers": [
            "Development Status :: 5 - Production/Stable",
            "Intended Audience :: Information Technology",
            "Intended Audience :: Science/Research",
            "License :: OSI Approved :: GNU General Public License v2 (GPLv2)",
            "Programming Language :: Python :: 3",
            "Topic :: Scientific/Engineering :: Artificial Intelligence",
            "Topic :: Text Processing"
        ],
        "description": "# Deep Learning Library: pydbm\n\n`pydbm` is Python library for building Restricted Boltzmann Machine(RBM), Deep Boltzmann Machine(DBM), Long Short-Term Memory Recurrent Temporal Restricted Boltzmann Machine(LSTM-RTRBM), and Shape Boltzmann Machine(Shape-BM). From the view points of functionally equivalents and structural expansions, this library also prototypes many variants such as Encoder/Decoder based on LSTM with an Attention mechanism, Convolutional Auto-Encoder, ConvLSTM, and Spatio-temporal Auto-Encoder.\n\nSee also ...\n\n- [Automatic Summarization Library: pysummarization](https://github.com/chimera0/accel-brain-code/tree/master/Automatic-Summarization)\n   * If you want to implement the Sequence-to-Sequence(Seq2Seq) model for the automatic summarization by using `pydbm` to build the Encoder/Decoder controllers.\n- [Reinforcement Learning Library: pyqlearning](https://github.com/chimera0/accel-brain-code/tree/master/Reinforcement-Learning)\n    * If you want to implement the Deep Reinforcement Learning, especially for Deep Q-Network and Multi-agent Deep Q-Network by using `pydbm` as a Function Approximator.\n- [Generative Adversarial Networks Library: pygan](https://github.com/chimera0/accel-brain-code/tree/master/Generative-Adversarial-Networks)\n    * If you want to implement Generative Adversarial Networks(GANs) and Adversarial Auto-Encoders(AAEs) by using `pydbm` as components for Generative models based on the Statistical machine learning problems.\n- [Algorithmic Composition or Automatic Composition Library: pycomposer](https://github.com/chimera0/accel-brain-code/tree/master/Algorithmic-Composition)\n   * If you want to implement the Algorithmic Composer based on Generative Adversarial Networks(GANs) by using `pydbm` and `pygan` as components for Generative models based on the Statistical machine learning problems.\n\n## Documentation\n\nFull documentation is available on [https://code.accel-brain.com/Deep-Learning-by-means-of-Design-Pattern/](https://code.accel-brain.com/Deep-Learning-by-means-of-Design-Pattern/) . This document contains information on functionally reusability, functional scalability and functional extensibility.\n\n## Installation\n\nInstall using pip:\n\n```sh\npip install \"git+https://github.com/chimera0/accel-brain-code.git#egg=pydbm&subdirectory=Deep-Learning-by-means-of-Design-Pattern\"\n```\n\nOr, after cloning this repository, you can install this library with the following command:\n\n```bash\ngit clone https://github.com/chimera0/accel-brain-code.git\ncd Deep-Learning-by-means-of-Design-Pattern/\npython setup.py bdist_wheel\npip install dist/pydbm-{version}-{hoge}.whl\n```\n\n### Source code\n\nThe source code is currently hosted on GitHub.\n\n- [accel-brain-code/Deep-Learning-by-means-of-Design-Pattern](https://github.com/chimera0/accel-brain-code/tree/master/Deep-Learning-by-means-of-Design-Pattern)\n\n### Python package index(PyPI)\n\nInstallers for the latest released version are available at the Python package index.\n\n- [pydbm : Python Package Index](https://pypi.python.org/pypi/pydbm)\n\n### Dependencies\n\n- numpy: v1.13.3 or higher.\n- cython: v0.27.1 or higher.\n\n#### Options\n\n- Pillow(PIL): v5.0.0 or higher.\n    * Only if you want to use `ImageGenerator`.\n\n## Description\n\nThe function of `pydbm` is building and modeling **Restricted Boltzmann Machine**(RBM) and **Deep Boltzmann Machine**(DBM). The models are functionally equivalent to **stacked auto-encoder**. The basic function is the same as **dimensions reduction**(or **pre-learning**). And this library enables you to build many functional extensions from RBM and DBM such as Recurrent Temporal Restricted Boltzmann Machine(RTRBM), Recurrent Neural Network Restricted Boltzmann Machine(RNN-RBM), Long Short-Term Memory Recurrent Temporal Restricted Boltzmann Machine(LSTM-RTRBM), and Shape Boltzmann Machine(Shape-BM).\n\nAs more usecases, **RTRBM**, **RNN-RBM**, and **LSTM-RTRBM** can learn dependency structures in temporal patterns such as music, natural sentences, and n-gram. RTRBM is a probabilistic time-series model which can be viewed as a temporal stack of RBMs, where each RBM has a contextual hidden state that is received from the previous RBM and is used to modulate its hidden units bias. The RTRBM can be understood as a sequence of conditional RBMs whose parameters are the output of a deterministic RNN, with the constraint that the hidden units must describe the conditional distributions. This constraint can be lifted by combining a full RNN with distinct hidden units. In terms of this possibility, RNN-RBM and LSTM-RTRBM are structurally expanded model from RTRBM that allows more freedom to describe the temporal dependencies involved.\n\nThe usecases of **Shape-BM** are image segmentation, object detection, inpainting and graphics. Shape-BM is the model for the task of modeling binary shape images, in that samples from the model look realistic and it can generalize to generate samples that differ from training examples.\n\n<table border=\"0\">\n    <tr>\n        <td>\n            <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/horse099.jpg\" />\n        <p>Image in <a href=\"https://avaminzhang.wordpress.com/2012/12/07/%E3%80%90dataset%E3%80%91weizmann-horses/\" target=\"_blank\">the Weizmann horse dataset</a>.</p>\n        </td>\n        <td>\n            <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/horse099_binary.png\" />\n \u00a0 \u00a0 \u00a0 \u00a0 \u00a0 \u00a0<p>Binarized image.</p>\n \u00a0 \u00a0 \u00a0 \u00a0</td>\n        <td>\n            <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/reconstructed_horse099.gif\" />\n            <p>Reconstructed image by Shape-BM.</p>\n        </td>\n    </tr>\n</table>\n\n### The structure of RBM.\n\nAccording to graph theory, the structure of RBM corresponds to a complete bipartite graph which is a special kind of bipartite graph where every node in the visible layer is connected to every node in the hidden layer. Based on statistical mechanics and thermodynamics(Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. 1985), the state of this structure can be reflected by the energy function:\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/energy_function_of_rbm.png\" /></div>\n\nwhere <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/b.png\" /> is a bias in visible layer, <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/c.png\" /> is a bias in hidden layer, <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/v.png\" /> is an activity or a state in visible layer, <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/h.png\" /> is an activity or a state in hidden layer, and <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/w.png\" /> is a weight matrix in visible and hidden layer. The activities can be calculated as the below product, since the link of activations of visible layer and hidden layer are conditionally independent.\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/conditionally_independent.png\" /></div>\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/define_lambda.png\" /></div>\n\n### The learning equations of RBM.\n\nBecause of the rules of conditional independence, the learning equations of RBM can be introduced as simple form. The distribution of visible state <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/v.png\" /> which is marginalized over the hidden state <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/h.png\" /> is as following:\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/distribution_of_visible_state.png\" /></div>\n\nwhere <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/partition_function.png\" /> is a partition function in statistical mechanics or thermodynamics. Let <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/D.png\" /> be set of observed data points, then <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/D_defined.png\" />. Therefore the gradients on the parameter <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/theta.png\" /> of the log-likelihood function are\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/gradients_v_on_the_parameter_theta.png\" /></div>\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/gradients_h_on_the_parameter_theta.png\" /></div>\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/gradients_w_on_the_parameter_theta.png\" /></div>\n\nwhere <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/expected_value.png\" /> is an expected value for <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/p_x_theta.png\" />. <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/sig.png\" /> is a sigmoid function.\n\nThe learning equations of RBM are introduced by performing control so that those gradients can become zero.\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/learning_equation_of_RBM.png\" /></div>\n\n### Contrastive Divergence as an approximation method.\n\nIn relation to RBM, **Contrastive Divergence**(CD) is a method for approximation of the gradients of the log-likelihood(Hinton, G. E. 2002). The procedure of this method is similar to Markov Chain Monte Carlo method(MCMC). However, unlike MCMC, the visbile variables to be set first in visible layer is not randomly initialized but the observed data points in training dataset are set to the first visbile variables. And, like Gibbs sampler, drawing samples from hidden variables and visible variables is repeated `k` times. Empirically (and surprisingly), `k` is considered to be `1`.\n\n### The structure of DBM.\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/image_dbn_and_dbm.png\" />\n<p>Salakhutdinov, R., Hinton, G. E. (2009). Deep boltzmann machines. In International conference on artificial intelligence and statistics (pp. 448-455). p451.</p>\n</div>\n\nAs is well known, DBM is composed of layers of RBMs stacked on top of each other(Salakhutdinov, R., & Hinton, G. E. 2009). This model is a structural expansion of Deep Belief Networks(DBN), which is known as one of the earliest models of Deep Learning(Le Roux, N., & Bengio, Y. 2008). Like RBM, DBN places nodes in layers. However, only the uppermost layer is composed of undirected edges, and the other consists of directed edges. DBN with `R` hidden layers is below probabilistic model:\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/dbn_model.png\" /></div>\n\nwhere `r = 0` points to visible layer. Considerling simultaneous distribution in top two layer, \n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/dbn_top_two_layer_joint.png\" /></div>\n\nand conditional distributions in other layers are as follows:\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/dbn_other_layers.png\" /></div>\n\nThe pre-learning of DBN engages in a procedure of recursive learning in layer-by-layer. However, as you can see from the difference of graph structure, DBM is slightly different from DBN in the form of pre-learning. For instance, if `r = 1`, the conditional distribution of visible layer is \n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/conditional_distribution_of_visible_layer.png\" />.</div>\n\nOn the other hand, the conditional distribution in the intermediate layer is\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/dbm_other_hidden_layer.png\" /></div>\n\nwhere `2` has been introduced considering that the intermediate layer `r` receives input data from Shallower layer\n`r-1` and deeper layer `r+1`. DBM sets these parameters as initial states.\n\n### DBM as a Stacked Auto-Encoder.\n\nDBM is functionally equivalent to a **Stacked Auto-Encoder**, which is-a neural network that tries to reconstruct its input. To *encode* the observed data points, the function of DBM is as linear transformation of feature map below\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/dbm_encoder.png\" />.</div>\n\nOn the other hand, to *decode* this feature points, the function of DBM is as linear transformation of feature map below\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/dbm_decoder.png\" />.</div>\n\nThe reconstruction error should be calculated in relation to problem setting. This library provides a default method, which can be overridden, for error function that computes Mean Squared Error(MSE). For instance, my jupyter notebook: [demo/demo_stacked_auto_encoder.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_stacked_auto_encoder.ipynb) demonstrates the reconstruction errors of DBM which is a Stacked Auto-Encoder.\n\n### Structural expansion for RTRBM.\n\nThe **RTRBM** (Sutskever, I., et al. 2009) is a probabilistic time-series model which can be viewed as a temporal stack of RBMs, where each RBM has a contextual hidden state that is received from the previous RBM and is used to modulate its hidden units bias. Let <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/previous_step_h.png\" /> be the hidden state in previous step `t-1`. The conditional distribution in hidden layer in time `t` is \n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/rtrbm_model.png\" /></div>\n\nwhere <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/W_R.png\" /> is weight matrix in each time steps. Then sampling of observed data points is is as following:\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/rtrbm_prob_model.png\" /></div>\n\nWhile the hidden units are binary during inference and sampling, it is the mean-field value <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/hat_h_t.png\" /> that is transmitted to its successors.\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/hat_h_rtrbm.png\" /></div>\n\n### Structural expansion for RNN-RBM.\n\nThe RTRBM can be understood as a sequence of conditional RBMs whose parameters are the output of a deterministic RNN, with the constraint that the hidden units must describe the conditional distributions and convey temporal information. This constraint can be lifted by combining a full RNN with distinct hidden units. **RNN-RBM** (Boulanger-Lewandowski, N., et al. 2012), which is the more structural expansion of RTRBM, has also hidden units <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/hat_h_t.png\" />.\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/rtrbm_and_rnn-rbm.png\" />\n<p>Boulanger-Lewandowski, N., Bengio, Y., & Vincent, P. (2012). Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. arXiv preprint arXiv:1206.6392., p4.</p>\n<p>Single arrows represent a deterministic function, double arrows represent the stochastic hidden-visible connections of an RBM.</p>\n</div>\n\nThe biases are linear function of <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/hat_h_t.png\" />. This hidden units are only connected to their direct predecessor <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/hat_h_t_1.png\" /> and visible units in time `t` by the relation:\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/hat_h_relation.png\" /></div>\n\nwhere <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/W_2.png\" /> and <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/W_3.png\" /> are weight matrixes.\n\n### Structural expansion for LSTM-RTRBM.\n\nAn example of the application to polyphonic music generation(Lyu, Q., et al. 2015) clued me in on how is it possible to connect RTRBM with LSTM.\n\n#### Structure of LSTM.\n\nOriginally, Long Short-Term Memory(LSTM) networks as a special RNN structure has proven stable and\npowerful for modeling long-range dependencies. The Key point of structural expansion is its memory cell <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/c_t.png\" /> which essentially acts as an accumulator of the state information. Every time observed data points are given as new information <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/g_t.png\" /> and input to LSTM's input gate, its information will be accumulated to the cell if the input gate <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/i_t.png\" /> is activated. The past state of cell <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/c_t-1.png\" /> could be forgotten in this process if LSTM's forget gate <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/f_t.png\" /> is on. Whether the latest cell output <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/c_t.png\" /> will be propagated to the final state <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/h_t.png\" /> is further controlled by the output gate <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/o_t.png\" />.\n\nOmitting so-called peephole connection, it makes possible to combine the activations in LSTM gates into an affine transformation below.\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/lstm_affine.png\" /></div>\n\nwhere <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/W_lstm.png\" /> is a weight matrix which connects observed data points and hidden units in LSTM gates, and <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/u.png\" /> is a weight matrix which connects hidden units as a remembered memory in LSTM gates. Furthermore, activation functions are as follows:\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/lstm_given.png\" /></div>\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/lstm_input_gate.png\" /></div>\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/lstm_forget_gate.png\" /></div>\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/lstm_output_gate.png\" /></div>\n\nand the acitivation of memory cell and hidden units are calculated as follows:\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/lstm_memory_cell.png\" /></div>\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/lstm_hidden_activity.png\" /></div>\n\n#### Structure of LSTM-RTRBM.\n\n**LSTM-RTRBM** model integrates the ability of LSTM in memorizing and retrieving useful history information, together with the advantage of RBM in high dimensional data modelling(Lyu, Q., Wu, Z., Zhu, J., & Meng, H. 2015, June). Like RTRBM, LSTM-RTRBM also has the recurrent hidden units. Let <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/pre_hidden_units.png\" /> be previous hidden units. The conditional distribution of the current hidden layer is as following:\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/LSTM-RTRBM_current_hidden_distribution.png\" /></div>\n\nwhere <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/W_R.png\" /> is a weight matrix which indicates the connectivity between states at each time step in RBM. Now, sampling the observed data points <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/v_T.png\" /> in RTRBM is as follows.\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/LSTM-RTRBM_sampling_v_T.png\" /></div>\n\n> \"Adding LSTM units to RTRBM is not trivial, considering RTRBM\u2019s hidden units and visible units are intertwined in inference and learning. The simplest way to circumvent this difficulty is to use bypass connections from LSTM units to the hidden units besides the existing recurrent connections of hidden units, as in LSTM-RTRBM.\"\n<div>Lyu, Q., Wu, Z., & Zhu, J. (2015, October). Polyphonic music modelling with LSTM-RTRBM. In Proceedings of the 23rd ACM international conference on Multimedia (pp. 991-994). ACM., p.993.</div>\n\nTherefore it is useful to introduce a distinction of *channel* which means the sequential information. <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/W_R.png\" /> indicates the direct connectivity in RBM, while <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/W_L.png\" /> can be defined as a concept representing the previous time step combination in the LSTM units. Let <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/h_R.png\" /> and <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/h_R.png\" /> be the hidden units indicating short-term memory and long-term memory, respectively. Then sampling the observed data points <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/v_T.png\" /> in LSTM-RTRBM can be re-described as follows.\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/LSTM-RTRBM_sampling_v_T_redescribed.png\" /></div>\n\n### Structural expansion for Shape-BM.\n\nThe concept of **Shape Boltzmann Machine** (Eslami, S. A., et al. 2014) provided inspiration to this library. This model uses below has two layers of hidden variables: <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/h_1.png\" /> and <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/h_2.png\" />. The visible units `v` arethe pixels of a binary image of size <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/N_times_M.png\" />. In the visible layer we enforce local receptive fields by connecting each hidden unit in <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/h_1.png\" /> only to a subset of the visible units, corresponding to one of four rectangular patches. In order to encourage boundary consistency each patch overlaps its neighbor by <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/r.png\" /> pixels and so has side lengths of <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/n_2_r_2.png\" /> and <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/m_2_r_2.png\" />. In this model, the weight matrix in visible and hidden layer correspond to conectivity between the four sets of hidden units and patches, however the visible biases <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/b_i.png\" /> are not shared.\n\n<div align=\"center\">\n <table style=\"border: none;\">\n  <tr>\n   <td width=\"45%\" align=\"center\">\n        <div>\n        <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/shape-bm-3d.png\" />\n        <p>Eslami, S. A., Heess, N., Williams, C. K., & Winn, J. (2014). The shape boltzmann machine: a strong model of object shape. International Journal of Computer Vision, 107(2), 155-176., p156.</p>\n        </div>\n   </td>\n   <td width=\"45%\" align=\"center\">\n        <div>\n        <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/sbm_network.png\" />\n        <p>Eslami, S. A., Heess, N., Williams, C. K., & Winn, J. (2014). The shape boltzmann machine: a strong model of object shape. International Journal of Computer Vision, 107(2), 155-176., p156.</p>\n        </div>\n   </td>\n  </tr>\n </table>\n</div>\n\nThe Shape-BM is a DBM in three layer. The learning algorithm can be completed by optimization of\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/sbm_prob.png\" /></div>\n\nwhere <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/sbm_params.png\" />.\n\n### The Commonality/Variability Analysis in order to practice object-oriented design.\n\nFrom perspective of *commonality/variability analysis* in order to practice object-oriented design, the concepts of RBM and DBM paradigms can be organized as follows:\n\n<img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/draw.io/pydbm_dbm_class_diagram.png?1\" />\n\nPay attention to the interface of the above class diagram. While each model is *common* in that it is constituted by stacked RBM, its approximation methods and activation functions are *variable* depending on the problem settings.\n\nConsidering the *commonality*, it is useful to design based on `Builder Pattern` represented by `DBMBuilder` or `RTRBMBuilder`, which separates the construction of RBM object `RestrictedBoltzmannMachine` from its representation by `DBMDirector` or `RTRBMDirector` so that the same construction process can create different representations such as DBM, RTRBM, RNN-RBM, and Shape-BM. Additionally, the models of all neural networks are *common* in that they possess like synapses by obtaining computation graphs without exception. So the class `Synapse` is contained in various models in a state where computation graphs of weight matrix and bias vector are held in the field.\n\nOn the other hand, to deal with the *variability*, `Strategy Pattern`, which provides a way to define a family of algorithms such as approximation methods implemented by inheriting the interface `ApproximateInterface`, and also activation functions implemented by inheriting the interface `ActivatingFunctionInterface`, is useful design method, which is encapsulate each one as an object, and make them interchangeable from the point of view of functionally equivalent. `Template Method Pattern` is also useful design method to design the optimizer in this library because this design pattern makes it possible to define the skeleton of an algorithm in a parameter tuning, deferring some steps to client subclasses such as `SGD`, `AdaGrad`, `RMSProp`, `NAG`,  `Adam` or `Nadam`. Template Method lets subclasses redefine certain steps of an algorithm without changing the algorithm\u2019s structure.\n\n### Functionally equivalent: Encoder/Decoder based on LSTM.\n\nThe methodology of *equivalent-functionalism* enables us to introduce more functional equivalents and compare problem solutions structured with different algorithms and models in common problem setting. For example, in dimension reduction problem, the function of **Encoder/Decoder schema** is equivalent to **DBM** as a **Stacked Auto-Encoder**.\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/latex/encoder_decoder.png\" />\n<p>Cho, K., Van Merri\u00ebnboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078., p2.</p>\n</div>\n\nAccording to the neural networks theory, and in relation to manifold hypothesis, it is well known that multilayer neural networks can learn features of observed data points and have the feature points in hidden layer. High-dimensional data can be converted to low-dimensional codes by training the model such as **Stacked Auto-Encoder** and **Encoder/Decoder** with a small central layer to reconstruct high-dimensional input vectors. This function of dimensionality reduction facilitates feature expressions to calculate similarity of each data point.\n\nThis library provides **Encoder/Decoder based on LSTM**, which is a reconstruction model and makes it possible to extract series features embedded in deeper layers. The LSTM encoder learns a fixed length vector of time-series observed data points and the LSTM decoder uses this representation to reconstruct the time-series using the current hidden state and the value inferenced at the previous time-step.\n\n<img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/draw.io/pydbm_rnn_class_diagram.png\" />\n\nAs in the above class diagram, in this library, the class `EncoderDecoderController` can be composed of two `LSTMModel`s. `LSTMModel` is-a `ReconstructableModel`, which has a learning method and an inference method like the ordinary supervised learning model.\n\nAn example is illustrated in this my jupyter notebook: [demo/demo_sine_wave_prediction_by_LSTM_encoder_decoder.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_sine_wave_prediction_by_LSTM_encoder_decoder.ipynb). This notebook demonstrates a simple sine wave prediction by Encoder/Decoder based on LSTM.\n\n#### Encoder/Decoder for Anomaly Detection(EncDec-AD)\n\nOne interesting application example is the **Encoder/Decoder for Anomaly Detection (EncDec-AD)** paradigm (Malhotra, P., et al. 2016). This reconstruction model learns to reconstruct *normal* time-series behavior, and thereafter uses reconstruction error to detect anomalies. Malhotra, P., et al. (2016) showed that EncDec-AD paradigm is robust and can detect anomalies from predictable, unpredictable, periodic, aperiodic, and quasi-periodic time-series. Further, they showed that the paradigm is able to detect anomalies from short time-series (length as small as 30) as well as long time-series (length as large as 500).\n\nAs the prototype is exemplified in [demo/demo_anomaly_detection_by_enc_dec_ad.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_anomaly_detection_by_enc_dec_ad.ipynb), this library provides Encoder/Decoder based on LSTM as a EncDec-AD scheme.\n\n### Functionally equivalent: Convolutional Auto-Encoder.\n\n**Shape-BM** is a kind of problem solution in relation to problem settings such as image segmentation, object detection, inpainting and graphics. In this problem settings, **Convolutional Auto-Encoder**(Masci, J., et al., 2011) is a functionally equivalent of **Shape-BM**. A stack of Convolutional Auto-Encoder forms a convolutional neural network(CNN), which are among the most successful models for supervised image classification. Each Convolutional Auto-Encoder is trained using conventional on-line gradient descent without additional regularization terms.\n\n<table border=\"0\">\n    <tr>\n        <td>\n            <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/horse099.jpg\" />\n        <p>Image in <a href=\"https://avaminzhang.wordpress.com/2012/12/07/%E3%80%90dataset%E3%80%91weizmann-horses/\" target=\"_blank\">the Weizmann horse dataset</a>.</p>\n        </td>\n        <td>\n            <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/reconstructed_horse099.gif\" />\n            <p>Reconstructed image by <strong>Shape-BM</strong>.</p>\n        </td>\n        <td>\n            <img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/reconstructed_by_CAE.gif\" />\n            <p>Reconstructed image by <strong>Convolutional Auto-Encoder</strong>.</p>\n        </td>\n    </tr>\n</table>\n\nMy jupyter notebook: [demo/demo_convolutional_auto_encoder.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_convolutional_auto_encoder.ipynb) also demonstrates various reconstructed images.\n\nThis library can draw a distinction between **Stacked Auto-Encoder** and **Convolutional Auto-Encoder**, and is able to design and implement respective models. **Stacked Auto-Encoder** ignores the 2 dimentional image structures. In many cases, the rank of observed tensors extracted from image dataset is more than 3. This is not only a problem when dealing with realistically sized inputs, but also introduces redundancy in the parameters, forcing each feature to be global. Like **Shape-BM**, **Convolutional Auto-Encoder** differs from **Stacked Auto-Encoder** as their weights are shared among all locations in the input, preserving spatial locality. Hence, the reconstructed image data is due to a linear combination of basic image patches based on the latent code.\n\nIn this library, **Convolutional Auto-Encoder** is also based on **Encoder/Decoder** scheme. The *encoder* is to the *decoder* what the *Convolution* is to the *Deconvolution*. The Deconvolution also called transposed convolutions \"work by swapping the forward and backward passes of a convolution.\" (Dumoulin, V., & Visin, F. 2016, p20.)\n\n### Structural expansion for Convolutional LSTM(ConvLSTM).\n\n**Convolutional LSTM(ConvLSTM)**(Xingjian, S. H. I. et al., 2015), which is a model that structurally couples **convolution operators** to **LSTM** networks, can be utilized as components in constructing the Encoder/Decoder. The ConvLSTM is suitable for spatio-temporal data due to its inherent convolutional structure. \n\n<div>\n<img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/model_of_conv_lstm.png\" />\n<p><cite>Xingjian, S. H. I., Chen, Z., Wang, H., Yeung, D. Y., Wong, W. K., & Woo, W. C. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems (pp. 802-810), p806.</cite></p>\n</div>\n\nThis library also makes it possible to build **Encoder/Decoder based on ConvLSTM**. My jupyter notebook: [demo/demo_conv_lstm.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_conv_lstm.ipynb) demonstrates that the Encoder/Decoder based on Convolutional LSTM(ConvLSTM) can learn images and reconstruct its.\n\n### Structural expansion for Spatio-Temporal Auto-Encoder.\n\n**Encoder/Decoder based on ConvLSTM** and **Convolutional Auto-Encoder** have a functional reusability to extend the structures to **Spatio-Temporal Auto-Encoder**, which can learn the regular patterns in the training videos(Baccouche, M., et al., 2012, Patraucean, V., et al. 2015). This model consists of spatial Auto-Encoder and temporal Encoder/Decoder. The spatial Auto-Encoder is a Convolutional Auto-Encoder for learning spatial structures of each video frame. The temporal Encoder/Decoder is an Encoder/Decoder based on LSTM scheme for learning temporal patterns of the encoded spatial structures. The spatial encoder and decoder have two convolutional and deconvolutional layers respectively, while the temporal encoder and decoder are to act as a twin LSTM models.\n\n<div><img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/spatio_temporal_auto_encoder_model.png\" />\n<p>Chong, Y. S., & Tay, Y. H. (2017, June). Abnormal event detection in videos using spatiotemporal autoencoder. In International Symposium on Neural Networks (pp. 189-196). Springer, Cham., p.195.</p>\n</div>\n\nBecause of the structural expansions, **ConvLSTM** and **Spatio-Temporal Auto-Encoder** can be consisted by `cnn` subpackage, which is responsible for convolution and deconvolution of spatial features, and `rnn` subpackage for controlling reconstruction of temporal features as in the following class diagram.\n\n<img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/draw.io/pydbm_cnn_class_diagram.png\" />\n\nIn `cnn` subpackage, the class `LayerableCNN` is an abstract class to implement CNN layers such as `ConvolutionLayer` and `MaxPoolingLayer`. `ConvolutionalAutoEncoder` and `SpatioTemporalAutoEncoder` have those CNN layers, especially `ConvolutionLayer` to convolve as forward propagation and to deconvolve as back propagation, and are common in the sense that each class has a learning method and an inference method. The difference is that only `SpatioTemporalAutoEncoder` is related to `ReconstructableModel` such as `LSTMModel` and `ConvLSTMModel` in `rnn` subpackage.\n\n#### Video recognition and reconstruction of video images.\n\n[demo/demo_spatio_temporal_auto_encoder.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_spatio_temporal_auto_encoder.ipynb) is a jupyter notebook which demonstrates the video recognition and reconstruction of video images by the Spatio-Temporal Auto-Encoder.\n\n### Structural extension from Auto-Encoders and Encoder/Decoders to energy-based models and Generative models.\n\nAuto-Encoders, such as the Convolutional Auto-Encoder, the Spatio-Temporal Auto-Encoder, and the DBM have in common that these models are Stacked Auto-Encoders. And the Encoder/Decoder based on LSTM or ConvLSTM share similarity with the RTRBM, RNN-RBM, and LSTM-RTRBM, as the reconstruction models. On the other hand, the Auto-Encoders and the Encoder/Decoders are not statistical mechanical energy-based models unlike with RBM or DBM.\n\nHowever, Auto-Encoders have traditionally been used to represent energy-based models. According to the statistical mechanical theory for energy-based models, Auto-Encoders constructed by neural networks can be associated with an energy landscape, akin to negative log-probability in a probabilistic model, which measures how well the Auto-Encoder can represent regions in the input space. The energy landscape has been commonly inferred heuristically, by using a training criterion that relates the Auto-Encoder to a probabilistic model such as a RBM. The energy function is identical to the free energy of the corresponding RBM, showing that Auto-Encoders and RBMs may be viewed as two different ways to derive training criteria for forming the same type of analytically defined energy landscape.\n\nThe view of the Auto-Encoder as a dynamical system allows us to understand how an energy function may be derived for the Auto-Encoder. This makes it possible to assign energies to Auto-Encoders with many different types of activation functions and outputs, and consider minimanization of reconstruction errors as energy minimanization(Kamyshanska, H., & Memisevic, R., 2014).\n\nWhen trained with some regularization terms, the Auto-Encoders have the ability to learn an energy manifold without supervision or negative examples(Zhao, J., et al., 2016). This means that even when an energy-based Auto-Encoding model is trained to reconstruct a real sample, the model contributes to discovering the data manifold by itself.\n\nThis library provides energy-based Auto-Encoders such as Contractive Convolutional Auto-Encoder(Rifai, S., et al., 2011), Repelling Convolutional Auto-Encoder(Zhao, J., et al., 2016), Denoising Auto-Encoders(Bengio, Y., et al., 2013), and Ladder Networks(Valpola, H., 2015). But it is more usefull to redescribe the Auto-Encoders in the framework of Generative Adversarial Networks(GANs)(Goodfellow, I., et al., 2014) to make those models function as not only energy-based models but also Generative models. For instance, theory of an Adversarial Auto-Encoders(AAEs)(Makhzani, A., et al., 2015) and energy-based GANs(EBGANs)(Zhao, J., et al., 2016) enables us to turn Auto-Encoders into a Generative models which referes energy functions. If you want to implement GANs and AAEs by using `pydbm` as components for Generative models based on the Statistical machine learning problems, see [Generative Adversarial Networks Library: pygan](https://github.com/chimera0/accel-brain-code/tree/master/Generative-Adversarial-Networks).\n\n### Composition and Correspondence in this library\n\nTo summarize the information so far into one class diagram, the outline is as follows.\n\n<img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/draw.io/pydbm_all_subpackage_class_diagram.png\" />\n\nUnlike `dbm` subpackage, `rnn` subpackage and `cnn` subpackage have an association with the interface `ComputableLoss`. The subclass are Loss functions such as Mean Square Error(MSE) and Cross Entropy. The *function* of loss functions for `dbm` is included in the *function* of energy functions optimized to minimize cost in the interface `ApproximateInterface`.\n\n<a name=\"usecase_building_the_deep_boltzmann_machine_for_feature_extracting\"></a>\n## Usecase: Building the Deep Boltzmann Machine for feature extracting.\n\nImport Python and Cython modules based on Builder Pattern.\n\n```python\n# The `Client` in Builder Pattern\nfrom pydbm.dbm.deep_boltzmann_machine import DeepBoltzmannMachine\n# The `Concrete Builder` in Builder Pattern.\nfrom pydbm.dbm.builders.dbm_multi_layer_builder import DBMMultiLayerBuilder\n# Contrastive Divergence for function approximation.\nfrom pydbm.approximation.contrastive_divergence import ContrastiveDivergence\n```\n\nImport Python and Cython modules of activation functions.\n\n```python\n# Logistic Function as activation function.\nfrom pydbm.activation.logistic_function import LogisticFunction\n# Tanh Function as activation function.\nfrom pydbm.activation.tanh_function import TanhFunction\n# ReLu Function as activation function.\nfrom pydbm.activation.relu_function import ReLuFunction\n```\n\nImport Python and Cython modules of optimizers, and instantiate the objects.\n\n```python\n# Stochastic Gradient Descent(SGD) as optimizer.\nfrom pydbm.optimization.optparams.sgd import SGD\n\n# is-a `OptParams`.\nopt_params = SGD(\n    # Momentum.\n    momentum=0.9\n)\n```\n\nIf you want to use not Stochastic Gradient Descent(SGD) but Adam(Kingma, D. P., & Ba, J., 2014) optimizer, import `Adam` and instantiate it.\n\n```python\n# Adam as a optimizer.\nfrom pydbm.optimization.optparams.adam import Adam\n\n# is-a `OptParams`.\nopt_params = Adam(\n    # BETA 1.\n    beta_1=0.9,\n    # BETA 2.\n    beta_2=0.99\n)\n```\n\nSetup parameters of regularization. For instance, constraining (or scale down) weight vectors and the probability of dropout(Srivastava, N., Hinton, G., et al., 2014, Zaremba, W., et al., 2014) can be set as follows.\n\n```python\n# Regularization for weights matrix\n# to repeat multiplying the weights matrix and `0.9`\n# until $\\sum_{j=0}^{n}w_{ji}^2 < weight\\_limit$.\nopt_params.weight_limit = 1e+03\n\n# Probability of dropout.\nopt_params.dropout_rate = 0.5\n```\n\nInstantiate objects and call the method.\n\n```python\n# Contrastive Divergence for visible layer and first hidden layer.\nfirst_cd = ContrastiveDivergence(opt_params=opt_params)\n# Contrastive Divergence for first hidden layer and second hidden layer.\nsecond_cd = ContrastiveDivergence(opt_params=opt_params)\n\n# DBM\ndbm = DeepBoltzmannMachine(\n    # `Concrete Builder` in Builder Pattern,\n    # which composes three restricted boltzmann machines for building a deep boltzmann machine.\n    DBMMultiLayerBuilder(),\n    # Dimention in visible layer, hidden layer, and second hidden layer.\n    [train_arr.shape[1], 10, train_arr.shape[1]],\n    # Setting objects for activation function.\n    [ReLuFunction(), LogisticFunction(), TanhFunction()],\n    # Setting the object for function approximation.\n    [first_cd, second_cd], \n    # Setting learning rate.\n    learning_rate=0.05\n)\n\n# Execute learning.\ndbm.learn(\n    # `np.ndarray` of observed data points.\n    train_arr,\n     # If approximation is the Contrastive Divergence, this parameter is `k` in CD method.\n    training_count=1,\n    # Batch size in mini-batch training.\n    batch_size=200,\n    # if `r_batch_size` > 0, the function of `dbm.learn` is a kind of reccursive learning.\n    r_batch_size=-1 \n)\n```\n\nIf you do not want to execute the mini-batch training, the value of `batch_size` must be `-1`. And `r_batch_size` is also parameter to control the mini-batch training but is refered only in inference and reconstruction. If this value is more than `0`,  the inferencing is a kind of reccursive learning with the mini-batch training.\n\nAnd the feature points can be extracted by this method.\n\n```python\nfeature_point_arr = dbm.get_feature_point(layer_number=1)\n```\n\n<a name=\"usecase_extracting_all_feature_points_for_dimensions_reduction_or_pre_learning\"></a>\n## Usecase: Extracting all feature points for dimensions reduction(or pre-learning)\n\nImport Python and Cython modules and instantiate the objects in the same manner as <a href=\"#usecase_building_the_deep_boltzmann_machine_for_feature_extracting\">Usecase: Building the Deep Boltzmann Machine for feature extracting.</a>\n\nImport and instantiate not `DeepBoltzmannMachine` but `StackedAutoEncoder`, and call the method.\n\n```python\n# `StackedAutoEncoder` is-a `DeepBoltzmannMachine`.\nfrom pydbm.dbm.deepboltzmannmachine.stacked_auto_encoder import StackedAutoEncoder\n\n# is-a `DeepBoltzmannMachine`.\ndbm = StackedAutoEncoder(\n    DBMMultiLayerBuilder(),\n    [target_arr.shape[1], 10, target_arr.shape[1]],\n    activation_list,\n    approximaion_list,\n    learning_rate=0.05 # Setting learning rate.\n)\n\n# Execute learning.\ndbm.learn(\n    target_arr,\n    1, # If approximation is the Contrastive Divergence, this parameter is `k` in CD method.\n \u00a0 \u00a0batch_size=200,  # Batch size in mini-batch training.\n    r_batch_size=-1  # if `r_batch_size` > 0, the function of `dbm.learn` is a kind of reccursive learning.\n)\n```\n\nThe function of `computable_loss` is computing the reconstruction error. `MeanSquaredError` is-a `ComputableLoss`, which is so-called Loss function.\n\n### Extract reconstruction error rate.\n\nYou can check the reconstruction error rate. During the approximation of the Contrastive Divergence, the mean squared error(MSE) between the observed data points and the activities in visible layer is computed as the reconstruction error rate.\n\nCall `get_reconstruct_error_arr` method as follow.\n\n```python\nreconstruct_error_arr = dbm.get_reconstruct_error_arr(layer_number=0)\n```\n\n`layer_number` corresponds to the index of `approximaion_list`. And `reconstruct_error_arr` is the `np.ndarray` of reconstruction error rates.\n\n### Extract the result of dimention reduction\n\nAnd the result of dimention reduction can be extracted by this property.\n\n```python\npre_learned_arr = dbm.feature_points_arr\n```\n\n### Extract weights obtained by pre-learning. \n\nIf you want to get the pre-learning weights, call `get_weight_arr_list` method.\n\n```python\nweight_arr_list = dbm.get_weight_arr_list()\n```\n`weight_arr_list` is the `list` of weights of each links in DBM. `weight_arr_list[0]` is 2-d `np.ndarray` of weights between visible layer and first hidden layer.\n\n### Extract biases obtained by pre-learning.\n\nCall `get_visible_bias_arr_list` method and `get_hidden_bias_arr_list` method in the same way.\n\n```python\nvisible_bias_arr_list = dbm.get_visible_bias_arr_list()\nhidden_bias_arr_list = dbm.get_hidden_bias_arr_list()\n```\n\n`visible_bias_arr_list` and `hidden_bias_arr_list` are the `list` of biases of each links in DBM.\n\n### Save pre-learned parameters.\n\nThe object `dbm`, which is-a `DeepBoltzmannMachine`, has the method `save_pre_learned_params`, to store the pre-learned parameters in compressed <a href=\"https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.lib.format.html#module-numpy.lib.format\" target=\"_blank\">NPY format</a> files.\n\n```python\n# Save pre-learned parameters.\ndbm.save_pre_learned_params(\n    # Path of dir. If `None`, the file is saved in the current directory.\n    dir_path=\"/var/tmp/\",\n    # The naming rule of files. If `None`, this value is `dbm`.\n    file_name=\"demo_dbm\"\n)\n```\n\n### Transfer learning in DBM.\n\n`DBMMultiLayerBuilder` can be given `pre_learned_path_list` which is a `list` of file paths that store pre-learned parameters.\n\n```python\ndbm = StackedAutoEncoder(\n    DBMMultiLayerBuilder(\n        # `list` of file path that stores pre-learned parameters.\n        pre_learned_path_list=[\n            \"/var/tmp/demo_dbm_0.npz\",\n            \"/var/tmp/demo_dbm_1.npz\"\n        ]\n    ),\n    [next_target_arr.shape[1], 10, next_target_arr.shape[1]],\n    activation_list,\n    approximaion_list,\n    # Setting learning rate.\n    0.05\n)\n\n# Execute learning.\ndbm.learn(\n    next_target_arr,\n    1, # If approximation is the Contrastive Divergence, this parameter is `k` in CD method.\n \u00a0 \u00a0batch_size=200,  # Batch size in mini-batch training.\n    r_batch_size=-1  # if `r_batch_size` > 0, the function of `dbm.learn` is a kind of reccursive learning.\n)\n```\n\nIf you want to know how to minimize the reconstructed error, see my Jupyter notebook: [demo/demo_stacked_auto_encoder.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_stacked_auto_encoder.ipynb).\n\n### Performance\n\nRun a program: [test/demo_stacked_auto_encoder.py](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/test/demo_stacked_auto_encoder.py)\n\n```sh\ntime python test/demo_stacked_auto_encoder.py\n```\n\nThe result is follow.\n \n```sh\n\nreal    1m59.875s\nuser    1m30.642s\nsys     0m29.232s\n```\n\n#### Detail\n\nThis experiment was performed under the following conditions.\n\n##### Machine type\n\n- vCPU: `2`\n- memory: `8GB`\n- CPU Platform: Intel Ivy Bridge\n\n##### Observation Data Points\n\nThe observated data is the result of `np.random.normal(loc=0.5, scale=0.2, size=(10000, 10000))`.\n\n##### Number of units\n\n- Visible layer: `10000`\n- hidden layer(feature point): `10`\n- hidden layer: `10000`\n\n##### Activation functions\n\n- visible:                Logistic Function\n- hidden(feature point):  Logistic Function\n- hidden:                 Logistic Function\n\n##### Approximation\n\n- Contrastive Divergence\n\n##### Hyper parameters\n\n- Learning rate: `0.05`\n- Dropout rate: `0.5`\n\n##### Feature points\n\n```\n[[0.092057   0.08856277 0.08699257 ... 0.09167331 0.08937846 0.0880063 ]\n [0.09090537 0.08669612 0.08995347 ... 0.08641837 0.08750935 0.08617442]\n [0.10187259 0.10633451 0.10060372 ... 0.10170306 0.10711189 0.10565192]\n ...\n [0.21540273 0.21737737 0.20949192 ... 0.20974982 0.2208562  0.20894371]\n [0.30749327 0.30964707 0.2850683  ... 0.29191507 0.29968456 0.29075691]\n [0.68022984 0.68454348 0.66431651 ... 0.67952715 0.6805653  0.66243178]]\n```\n\n##### Reconstruct error\n\n```\n [ 0.11668085 0.07513545 0.091044  ...,  0.0719339  0.07976882 0.09121697]\n```\n\n<a name=\"usecase_building_the_rtrbm_for_recursive_learning\"></a>\n## Usecase: Building the RTRBM for recursive learning.\n\nImport Python and Cython modules.\n\n```python\n# Logistic Function as activation function.\nfrom pydbm.activation.logistic_function import LogisticFunction\n# Tanh Function as activation function.\nfrom pydbm.activation.tanh_function import TanhFunction\n# Stochastic Gradient Descent(SGD) as optimizer.\nfrom pydbm.optimization.optparams.sgd import SGD\n# The `Client` in Builder Pattern for building RTRBM.\nfrom pydbm.dbm.recurrent_temporal_rbm import RecurrentTemporalRBM\n```\n\nInstantiate objects and execute learning.\n\n```python\n# The `Client` in Builder Pattern for building RTRBM.\nrt_rbm = RecurrentTemporalRBM(\n    # The number of units in visible layer.\n    visible_num=observed_arr.shape[-1],\n    # The number of units in hidden layer.\n    hidden_num=100,\n    # The activation function in visible layer.\n    visible_activating_function=TanhFunction(),\n    # The activation function in hidden layer.\n    hidden_activating_function=TanhFunction(),\n    # The activation function in RNN layer.\n    rnn_activating_function=LogisticFunction(),\n    # is-a `OptParams`.\n    opt_params=SGD(),\n    # Learning rate.\n    learning_rate=1e-05\n)\n```\n\n### Learning.\n\nThe `rt_rbm` has a `learn` method, to execute learning observed data points. This method can receive a `np.ndarray` of observed data points, which is a rank-3 array-like or sparse matrix of shape: (`The number of samples`, `The length of cycle`, `The number of features`), as the first argument.\n\n```python\n# Learning.\nrt_rbm.learn(\n    # The `np.ndarray` of observed data points.\n    observed_arr,\n    # Training count.\n    training_count=1000, \n    # Batch size.\n    batch_size=200\n)\n```\n\n### Inferencing.\n\nAfter learning, the `rt_rbm` provides a function of `inference` method. \n\n```python\n# Execute recursive learning.\ninferenced_arr = rt_rbm.inference(\n    test_arr,\n    training_count=1, \n    r_batch_size=-1\n)\n```\n\nThe shape of `test_arr` is equivalent to `observed_arr`. Returned value `inferenced_arr` is generated by input parameter `test_arr` and can be considered as a feature expression of `test_arr` based on the distribution of `observed_arr`. In other words, the features of `inferenced_arr` is a summary of time series information in `test_arr` and then the shape is rank-2 array-like or sparse matrix: (`The number of samples`, `The number of features`).\n\n### Feature points.\n\nOn the other hand, the `rt_rbm` has a `rbm` which also stores the feature points in hidden layers. To extract this embedded data, call the method as follows.\n\n```python\nfeature_points_arr = rt_rbm.rbm.get_feature_points()\n```\n\nThe shape of `feature_points_arr` is rank-2 array-like or sparse matrix: (`The number of samples`, `The number of units in hidden layers`). So this matrix also means time series data embedded as manifolds.\n\n### Reconstructed data.\n\nAlthough RTRBM itself is not an Auto-Encoder, it can be described as a reconstruction model. In this library, this model has an input reconstruction function.\n\n```python\nreconstructed_arr = rt_rbm.rbm.get_reconstructed_arr()\n```\n\nThe shape of `reconstructed_arr` is equivalent to `observed_arr`.\n\nIf you want to know how to measure its reconstruction errors, see my Jupyter notebook: [demo/demo_rt_rbm.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_rt_rbm.ipynb).\n\n### Save pre-learned parameters.\n\nThe object `rt_rbm`, which is-a `RecurrentTemporalRBM`, has the method `save_pre_learned_params`, to store the pre-learned parameters in a compressed <a href=\"https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.lib.format.html#module-numpy.lib.format\" target=\"_blank\">NPY format</a> file.\n\n```python\nrt_rbm.save_pre_learned_params(\"/var/tmp/demo_rtrbm.npz\")\n```\n\n### Transfer learning in RTRBM.\n\n`__init__` method of `RecurrentTemporalRBM` can be given `pre_learned_path_list` which is a `str` of file path that stores pre-learned parameters.\n\n```python\n# The `Client` in Builder Pattern for building RTRBM.\nrt_rbm = RecurrentTemporalRBM(\n    # The number of units in visible layer.\n    visible_num=observed_arr.shape[-1],\n    # The number of units in hidden layer.\n    hidden_num=100,\n    # The activation function in visible layer.\n    visible_activating_function=TanhFunction(),\n    # The activation function in hidden layer.\n    hidden_activating_function=TanhFunction(),\n    # The activation function in RNN layer.\n    rnn_activating_function=LogisticFunction(),\n    # is-a `OptParams`.\n    opt_params=SGD(),\n    # Learning rate.\n    learning_rate=1e-05,\n    # File path that stores pre-learned parameters.\n    pre_learned_path=\"/var/tmp/demo_rtrbm.npz\"\n)\n\n# Learning.\nrt_rbm.learn(\n    # The `np.ndarray` of observed data points.\n    observed_arr,\n    # Training count.\n    training_count=1000, \n    # Batch size.\n    batch_size=200\n)\n```\n\n## Usecase: Building the RNN-RBM for recursive learning.\n\nImport not `RecurrentTemporalRBM` but `RNNRBM`, which is-a `RecurrentTemporalRBM`.\n\n```python\n# The `Client` in Builder Pattern for building RNN-RBM.\nfrom pydbm.dbm.recurrenttemporalrbm.rnn_rbm import RNNRBM\n```\n\nInstantiate objects.\n\n```python\n# The `Client` in Builder Pattern for building RNN-RBM.\nrt_rbm = RNNRBM(\n    # The number of units in visible layer.\n    visible_num=observed_arr.shape[-1],\n    # The number of units in hidden layer.\n    hidden_num=100,\n    # The activation function in visible layer.\n    visible_activating_function=TanhFunction(),\n    # The activation function in hidden layer.\n    hidden_activating_function=TanhFunction(),\n    # The activation function in RNN layer.\n    rnn_activating_function=LogisticFunction(),\n    # is-a `OptParams`.\n    opt_params=SGD(),\n    # Learning rate.\n    learning_rate=1e-05\n)\n```\n\nThe function of learning, inferencing, saving pre-learned parameters, and transfer learning are equivalent to `rt_rbm` of RTRBM. See <a href=\"#usecase_building_the_rtrbm_for_recursive_learning\">Usecase: Building the RTRBM for recursive learning.</a>.\n\nIf you want to know how to measure its reconstruction errors, see my Jupyter notebook: [demo/demo_rnn_rbm.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_rnn_rbm.ipynb).\n\n## Usecase: Building the LSTM-RTRBM for recursive learning.\n\nImport not `RecurrentTemporalRBM` but `LSTMRTRBM`, which is-a `RecurrentTemporalRBM`.\n\n```python\n# The `Client` in Builder Pattern for building LSTM-RTRBM.\nfrom pydbm.dbm.recurrenttemporalrbm.lstm_rt_rbm import LSTMRTRBM\n```\n\nInstantiate objects.\n\n```python\n# The `Client` in Builder Pattern for building RNN-RBM.\nrt_rbm = LSTMRTRBM(\n    # The number of units in visible layer.\n    visible_num=observed_arr.shape[-1],\n    # The number of units in hidden layer.\n    hidden_num=100,\n    # The activation function in visible layer.\n    visible_activating_function=TanhFunction(),\n    # The activation function in hidden layer.\n    hidden_activating_function=TanhFunction(),\n    # The activation function in RNN layer.\n    rnn_activating_function=LogisticFunction(),\n    # is-a `OptParams`.\n    opt_params=SGD(),\n    # Learning rate.\n    learning_rate=1e-05\n)\n```\n\nThe function of learning, inferencing, saving pre-learned parameters, and transfer learning are equivalent to `rt_rbm` of RTRBM. See <a href=\"#usecase_building_the_rtrbm_for_recursive_learning\">Usecase: Building the RTRBM for recursive learning.</a>.\n\nIf you want to know how to measure its reconstruction errors, see my Jupyter notebook: [demo/demo_lstm_rt_rbm.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_lstm_rt_rbm.ipynb).\n\n\n## Usecase: Image segmentation by Shape-BM.\n\nFirst, acquire image data and binarize it.\n\n```python\nfrom PIL import Image\nimg = Image.open(\"horse099.jpg\")\nimg\n```\n\n<img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/horse099.jpg\" />\n\nIf you think the size of your image datasets may be large, resize it to an arbitrary size.\n\n```python\nimg = img.resize((255, 255))\n```\n\nConvert RGB images to binary images.\n\n```python\nimg_bin = img.convert(\"1\")\nimg_bin\n```\n\n<img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/horse099_binary.png\" />\n\nSet up hyperparameters.\n\n```python\nfilter_size = 5\noverlap_n = 4\n\nlearning_rate = 0.01\n```\n\n`filter_size` is the 'filter' size. This value must be more than `4`. And `overlap_n` is hyperparameter specific to Shape-BM. In the visible layer, this model has so-called local receptive fields by connecting each first hidden unit only to a subset of the visible units, corresponding to one of four square patches. Each patch overlaps its neighbor by `overlap_n` pixels (Eslami, S. A., et al, 2014).\n\n**Please note** that the recommended ratio of `filter_size` and `overlap_n` is 5:4. It is not a constraint demanded by pure theory of Shape Boltzmann Machine itself but is a kind of limitation to simplify design and implementation in this library. \n\nAnd import Python and Cython modules.\n\n```python\n# The `Client` in Builder Pattern\nfrom pydbm.dbm.deepboltzmannmachine.shape_boltzmann_machine import ShapeBoltzmannMachine\n# The `Concrete Builder` in Builder Pattern.\nfrom pydbm.dbm.builders.dbm_multi_layer_builder import DBMMultiLayerBuilder\n```\n\nInstantiate objects and call the method.\n\n```python\ndbm = ShapeBoltzmannMachine(\n    DBMMultiLayerBuilder(),\n    learning_rate=learning_rate,\n    overlap_n=overlap_n,\n    filter_size=filter_size\n)\n\nimg_arr = np.asarray(img_bin)\nimg_arr = img_arr.astype(np.float64)\n\n# Execute learning.\ndbm.learn(\n    # `np.ndarray` of image data.\n    img_arr,\n    # If approximation is the Contrastive Divergence, this parameter is `k` in CD method.\n    training_count=1,\n    # Batch size in mini-batch training.\n    batch_size=300,\n    # if `r_batch_size` > 0, the function of `dbm.learn` is a kind of reccursive learning.\n    r_batch_size=-1,\n    # Learning with the stochastic gradient descent(SGD) or not.\n    sgd_flag=True\n)\n```\n\nExtract `dbm.visible_points_arr` as the observed data points in visible layer. This `np.ndarray` is segmented image data.\n\n```python\ninferenced_data_arr = dbm.visible_points_arr.copy()\ninferenced_data_arr = 255 - inferenced_data_arr\nImage.fromarray(np.uint8(inferenced_data_arr))\n```\n\n<img src=\"https://storage.googleapis.com/accel-brain-code/Deep-Learning-by-means-of-Design-Pattern/img/reconstructed_09.png\" />\n\n### Save pre-learned parameters and transfer learning in Shape Boltzmann Machine.\n\nIn transfer learning problem setting, `ShapeBoltzmannMachine` is functionally equivalent to `StackedAutoEncoder`. See <a href=\"#usecase_extracting_all_feature_points_for_dimensions_reduction_or_pre_learning\">Usecase: Extracting all feature points for dimensions reduction(or pre-learning)</a>.\n\n## Usecase: Casual use by facade for building Encoder/Decoder based on LSTM.\n\nImport `facade` module for building Encoder/Decoder based on LSTM.\n\n```python\nfrom pydbm.rnn.facade_encoder_decoder import FacadeEncoderDecoder\n```\n\nIf you want to use an Attention mechanism, import `FacadeAttentionEncoderDecoder` instead.\n\n```python\nfrom pydbm.rnn.facade_attention_encoder_decoder import FacadeAttentionEncoderDecoder as FacadeEncoderDecoder\n```\n\nInstantiate object and call the method to learn observed data points.\n\n```python\n# `Facade` for casual user of Encoder/Decoder based on LSTM networks.\nfacade_encoder_decoder = FacadeEncoderDecoder(\n    # The number of units in input layers.\n    input_neuron_count=observed_arr.shape[-1],\n    # The length of sequences.\n    # This means refereed maxinum step `t` in feedforward.\n    seq_len=observed_arr.shape[1],\n    # Refereed maxinum step `t` in BPTT. If `0`, this class referes all past data in BPTT.\n    bptt_tau=observed_arr.shape[1],\n    # Verbose mode or not. If `True`, this class sets the logger level as `DEBUG`.\n    verbose_flag=True\n)\n```\n\nExecute learning.\n\n```python\nfacade_encoder_decoder.learn(\n    observed_arr=observed_arr,\n    target_arr=observed_arr\n)\n```\n\nThis method can receive a `np.ndarray` of observed data points, which is a rank-3 array-like or sparse matrix of shape: (`The number of samples`, `The length of cycle`, `The number of features`), as the first and second argument. If the value of this second argument is not equivalent to the first argument and the shape is (`The number of samples`, `The number of features`), in other words, the rank is 2, the function of `encoder_decoder_controller` corresponds to a kind of Regression model.\n\nAfter learning, the `facade_encoder_decoder` provides a function of `inference` method. \n\n```python\n# Execute recursive learning.\ninferenced_arr = facade_encoder_decoder.inference(test_arr)\n```\n\nThe shape of `test_arr` and `inferenced_arr` are equivalent to `observed_arr`. Returned value `inferenced_arr` is generated by input parameter `test_arr` and can be considered as a decoded data points based on encoded `test_arr`.\n\nOn the other hand, the `facade_encoder_decoder` also stores the feature points in hidden layers. To extract this embedded data, call the method as follows.\n\n```python\nfeature_points_arr = facade_encoder_decoder.get_feature_points()\n```\n\nThe shape of `feature_points_arr` is rank-2 array-like or sparse matrix: (`The number of samples`, `The number of units in hidden layers`). So this matrix also means time series data embedded as manifolds.\n\nYou can check the reconstruction error rate. Call `get_reconstruct_error` method as follow.\n\n```python\nreconstruct_error_arr = facade_encoder_decoder.get_reconstruction_error()\n```\n\nIf you want to know how to minimize the reconstructed error, see my Jupyter notebook: [demo/demo_sine_wave_prediction_by_LSTM_encoder_decoder.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_sine_wave_prediction_by_LSTM_encoder_decoder.ipynb).\n\n### Save pre-learned parameters.\n\nThe object `facade_encoder_decoder` has the method `save_pre_learned_params`, to store the pre-learned parameters in compressed <a href=\"https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.lib.format.html#module-numpy.lib.format\" target=\"_blank\">NPY format</a> files.\n\n```python\nfacade_encoder_decoder.save_pre_learned_params(\n    # File path that stores Encoder's parameters.\n    encoder_file_path=\"/var/tmp/encoder.npz\",\n    # File path that stores Decoder's parameters.\n    decoder_file_path=\"/var/tmp/decoder.npz\"\n)\n```\n\n### Transfer learning in Encoder/Decoder based on LSTM.\n\n`__init__` method of `FacadeEncoderDecoder` can be given `encoder_pre_learned_file_path` and `decoder_pre_learned_file_path`, which are `str` of file path that stores Encoder/Decoder's pre-learned parameters.\n\n```python\nfacade_encoder_decoder2 = FacadeEncoderDecoder(\n    # The number of units in input layers.\n    input_neuron_count=observed_arr.shape[-1],\n    # The length of sequences.\n    # This means refereed maxinum step `t` in feedforward.\n    seq_len=observed_arr.shape[1],\n    # Refereed maxinum step `t` in BPTT. If `0`, this class referes all past data in BPTT.\n    bptt_tau=observed_arr.shape[1],\n    # File path that stored Encoder's pre-learned parameters.\n    encoder_pre_learned_file_path=\"/var/tmp/encoder.npz\",\n    # File path that stored Decoder's pre-learned parameters.\n    decoder_pre_learned_file_path=\"/var/tmp/decoder.npz\",\n    # Verbose mode or not. If `True`, this class sets the logger level as `DEBUG`.\n    verbose_flag=True\n)\n\nfacade_encoder_decoder2.learn(\n    observed_arr=observed_arr,\n    target_arr=observed_arr\n)\n```\n\n### For more detail settings.\n\n`__init__` of `FacadeEncoderDecoder` can be given many parameters as follows.\n\n```python\n# `Facade` for casual user of Encoder/Decoder based on LSTM networks.\nfacade_encoder_decoder = FacadeEncoderDecoder(\n    # The number of units in input layers.\n    input_neuron_count=observed_arr.shape[-1],\n    # The number of units in hidden layers.\n    hidden_neuron_count=200,\n    # Epochs of Mini-batch.\n    epochs=200,\n    # Batch size of Mini-batch.\n    batch_size=20,\n    # Learning rate.\n    learning_rate=1e-05,\n    # Attenuate the `learning_rate` by a factor of this value every `attenuate_epoch`.\n    learning_attenuate_rate=0.1,\n    # Attenuate the `learning_rate` by a factor of `learning_attenuate_rate` every `attenuate_epoch`.\n    attenuate_epoch=50,\n    # Activation function in hidden layers.\n    hidden_activating_function=LogisticFunction(),\n    # Activation function in output layers.\n    output_activating_function=LogisticFunction(),\n    # Loss function.\n    computable_loss=MeanSquaredError(),\n    # Optimizer which is-a `OptParams`.\n    opt_params=Adam(),\n    # The length of sequences.\n    # This means refereed maxinum step `t` in feedforward.\n    seq_len=8,\n    # Refereed maxinum step `t` in Backpropagation Through Time(BPTT).\n    # If `0`, this class referes all past data in BPTT.\n    bptt_tau=8,\n    # Size of Test data set. If this value is `0`, the validation will not be executed.\n    test_size_rate=0.3,\n    # Tolerance for the optimization.\n    # When the loss or score is not improving by at least tol \n    # for two consecutive iterations, convergence is considered \n    # to be reached and training stops.\n    tol=0.0,\n    # Tolerance for deviation of loss.\n    tld=1.0,\n    # Verification function.\n    verificatable_result=VerificateFunctionApproximation(),\n    # Verbose mode or not. If `True`, this class sets the logger level as `DEBUG`.\n    verbose_flag=True\n)\n```\n\nIf you want to not only use casually the model but also hack it, see <a href=\"#usecase_build_encoder_decoder_based_on_LSTM_as_a_reconstruction_model\">Usecase: Build Encoder/Decoder based on LSTM as a reconstruction model.</a>.\n\n<a name=\"usecase_build_encoder_decoder_based_on_LSTM_as_a_reconstruction_model\"></a>\n## Usecase: Build Encoder/Decoder based on LSTM or ConvLSTM as a reconstruction model.\n\nConsider functionally reusability and possibility of flexible design, you should use not `FacadeEncoderDecoder` but `EncoderDecoderController` as follows.\n\nSetup logger for verbose output.\n\n```python\nfrom logging import getLogger, StreamHandler, NullHandler, DEBUG, ERROR\n\nlogger = getLogger(\"pydbm\")\nhandler = StreamHandler()\nhandler.setLevel(DEBUG)\nlogger.setLevel(DEBUG)\nlogger.addHandler(handler)\n```\n\nImport Python and Cython modules for computation graphs.\n\n```python\n# LSTM Graph which is-a `Synapse`.\nfrom pydbm.synapse.recurrenttemporalgraph.lstm_graph import LSTMGraph as EncoderGraph\nfrom pydbm.synapse.recurrenttemporalgraph.lstm_graph import LSTMGraph as DecoderGraph\n```\n\nIf you want to introduce the graph of decoder for building an Attention mechanism as the decoder, import `AttentionLSTMGraph` instead.\n\n```python\nfrom pydbm.synapse.recurrenttemporalgraph.lstmgraph.attention_lstm_graph import AttentionLSTMGraph as DecoderGraph\n```\n\nImport Python and Cython modules of activation functions.\n\n```python\n# Logistic Function as activation function.\nfrom pydbm.activation.logistic_function import LogisticFunction\n# Tanh Function as activation function.\nfrom pydbm.activation.tanh_function import TanhFunction\n```\n\nImport Python and Cython modules for loss function.\n\n```python\n# Loss function.\nfrom pydbm.loss.mean_squared_error import MeanSquaredError\n```\n\nImport Python and Cython modules for optimizer.\n\n```python\n# SGD as a optimizer.\nfrom pydbm.optimization.optparams.sgd import SGD as EncoderSGD\nfrom pydbm.optimization.optparams.sgd import SGD as DecoderSGD\n```\n\nIf you want to use not Stochastic Gradient Descent(SGD) but **Adam** optimizer, import `Adam`.\n\n```python\n# Adam as a optimizer.\nfrom pydbm.optimization.optparams.adam import Adam as EncoderAdam\nfrom pydbm.optimization.optparams.adam import Adam as DecoderAdam\n```\n\nFuthermore, import class for verification of function approximation.\n\n```python\n# Verification.\nfrom pydbm.verification.verificate_function_approximation import VerificateFunctionApproximation\n```\n\nThe activation by softmax function can be verificated by `VerificateSoftmax`.\n\n```python\nfrom pydbm.verification.verificate_softmax import VerificateSoftmax\n```\n\nAnd import LSTM Model and Encoder/Decoder schema.\n\n```python\n# LSTM model.\nfrom pydbm.rnn.lstm_model import LSTMModel as Encoder\nfrom pydbm.rnn.lstm_model import LSTMModel as Decoder\n\n# Encoder/Decoder\nfrom pydbm.rnn.encoder_decoder_controller import EncoderDecoderController\n```\n\nIf you want to build an Attention mechanism as the decoder, import `AttentionLSTMModel` instead.\n\n```python\nfrom pydbm.rnn.lstmmodel.attention_lstm_model import AttentionLSTMModel as Decoder\n```\n\nInstantiate `Encoder`.\n\n```python\n# Init.\nencoder_graph = EncoderGraph()\n\n# Activation function in LSTM.\nencoder_graph.observed_activating_function = TanhFunction()\nencoder_graph.input_gate_activating_function = LogisticFunction()\nencoder_graph.forget_gate_activating_function = LogisticFunction()\nencoder_graph.output_gate_activating_function = LogisticFunction()\nencoder_graph.hidden_activating_function = TanhFunction()\nencoder_graph.output_activating_function = LogisticFunction()\n\n# Initialization strategy.\n# This method initialize each weight matrices and biases in Gaussian distribution: `np.random.normal(size=hoge) * 0.01`.\nencoder_graph.create_rnn_cells(\n    input_neuron_count=observed_arr.shape[-1],\n    hidden_neuron_count=200,\n    output_neuron_count=1\n)\n\n# Optimizer for Encoder.\nencoder_opt_params = EncoderAdam()\nencoder_opt_params.weight_limit = 1e+03\nencoder_opt_params.dropout_rate = 0.5\n\nencoder = Encoder(\n    # Delegate `graph` to `LSTMModel`.\n    graph=encoder_graph,\n    # Refereed maxinum step `t` in BPTT. If `0`, this class referes all past data in BPTT.\n    bptt_tau=8,\n    # Size of Test data set. If this value is `0`, the validation will not be executed.\n    test_size_rate=0.3,\n    # Loss function.\n    computable_loss=MeanSquaredError(),\n    # Optimizer.\n    opt_params=encoder_opt_params,\n    # Verification function.\n    verificatable_result=VerificateFunctionApproximation(),\n    # Tolerance for the optimization.\n    # When the loss or score is not improving by at least tol \n    # for two consecutive iterations, convergence is considered \n    # to be reached and training stops.\n    tol=0.0\n)\n```\n\nInstantiate `Decoder`.\n\n```python\n# Init.\ndecoder_graph = DecoderGraph()\n\n# Activation function in LSTM.\ndecoder_graph.observed_activating_function = TanhFunction()\ndecoder_graph.input_gate_activating_function = LogisticFunction()\ndecoder_graph.forget_gate_activating_function = LogisticFunction()\ndecoder_graph.output_gate_activating_function = LogisticFunction()\ndecoder_graph.hidden_activating_function = TanhFunction()\ndecoder_graph.output_activating_function = LogisticFunction()\n\n# Initialization strategy.\n# This method initialize each weight matrices and biases in Gaussian distribution: `np.random.normal(size=hoge) * 0.01`.\ndecoder_graph.create_rnn_cells(\n    input_neuron_count=200,\n    hidden_neuron_count=200,\n    output_neuron_count=observed_arr.shape[-1]\n)\n\n# Optimizer for Decoder.\ndecoder_opt_params = DecoderAdam()\ndecoder_opt_params.weight_limit = 1e+03\ndecoder_opt_params.dropout_rate = 0.5\n\ndecoder = Decoder(\n    # Delegate `graph` to `LSTMModel`.\n    graph=decoder_graph,\n    # The length of sequences.\n    seq_len=8,\n    # Refereed maxinum step `t` in BPTT. If `0`, this class referes all past data in BPTT.\n    bptt_tau=8,\n    # Loss function.\n    computable_loss=MeanSquaredError(),\n    # Optimizer.\n    opt_params=decoder_opt_params,\n    # Verification function.\n    verificatable_result=VerificateFunctionApproximation(),\n    # Tolerance for the optimization.\n    # When the loss or score is not improving by at least tol \n    # for two consecutive iterations, convergence is considered \n    # to be reached and training stops.\n    tol=0.0\n)\n```\n\nInstantiate `EncoderDecoderController` and delegate `encoder` and `decoder` to this object.\n\n```python\nencoder_decoder_controller = EncoderDecoderController(\n    # is-a LSTM model.\n    encoder=encoder,\n    # is-a LSTM model.\n    decoder=decoder,\n    # The number of epochs in mini-batch training.\n    epochs=200,\n    # The batch size.\n    batch_size=100,\n    # Learning rate.\n    learning_rate=1e-05,\n    # Attenuate the `learning_rate` by a factor of this value every `attenuate_epoch`.\n    learning_attenuate_rate=0.1,\n    # Attenuate the `learning_rate` by a factor of `learning_attenuate_rate` every `attenuate_epoch`.\n    attenuate_epoch=50,\n    # Size of Test data set. If this value is `0`, the validation will not be executed.\n    test_size_rate=0.3,\n    # Loss function.\n    computable_loss=MeanSquaredError(),\n    # Verification function.\n    verificatable_result=VerificateFunctionApproximation(),\n    # Tolerance for the optimization.\n    # When the loss or score is not improving by at least tol \n    # for two consecutive iterations, convergence is considered \n    # to be reached and training stops.\n    tol=0.0\n)\n```\n\nIf you want to use ConvLSTM as `encoder` and `decoder`, instantiate `ConvLSTMModel` which is-a `LSTMModel` and is-a `ReconstructableModel`. See my jupyter notebook for details: [demo/demo_conv_lstm.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_conv_lstm.ipynb).\n\nIn any case, let's execute learning after instantiation is complete.\n\n```python\n# Learning.\nencoder_decoder_controller.learn(observed_arr, observed_arr)\n```\n\nIf you delegated `LSTMModel`s as `encoder` and `decoder`, this method can receive a `np.ndarray` of observed data points, which is a **rank-3 array-like or sparse matrix** of shape: (`The number of samples`, `The length of cycle`, `The number of features`), as the first and second argument. If the value of this second argument is not equivalent to the first argument and the shape is (`The number of samples`, `The number of features`), in other words, the rank is **2**, the function of `encoder_decoder_controller` corresponds to a kind of Regression model.\n\nOn the other hand, if you delegated `ConvLSTMModel`s as `encoder` and `decoder`, the rank of matrix is **5**. The shape is:(`The number of samples`, `The length of cycle`, `Channel`, `Height of images`, `Width of images`).\n\nAfter learning, the `encoder_decoder_controller` provides a function of `inference` method. \n\n```python\n# Execute recursive learning.\ninferenced_arr = encoder_decoder_controller.inference(test_arr)\n```\n\nThe shape of `test_arr` and `inferenced_arr` are equivalent to `observed_arr`. Returned value `inferenced_arr` is generated by input parameter `test_arr` and can be considered as a decoded data points based on encoded `test_arr`.\n\nOn the other hand, the `encoder_decoder_controller` also stores the feature points in hidden layers. To extract this embedded data, call the method as follows.\n\n```python\nfeature_points_arr = encoder_decoder_controller.get_feature_points()\n```\n\nIf `LSTMModel`s are delegated, the shape of `feature_points_arr` is **rank-3 array-like or sparse matrix**: (`The number of samples`, `The length of cycle`, `The number of units in hidden layers`). On the other hand, if `ConvLSTMModel`s are delegated, the shape of `feature_points_arr` is **rank-5 array-like or sparse matrix**:(`The number of samples`, `The length of cycle`, `Channel`, `Height of images`, `Width of images`). So the matrices also mean time series data embedded as manifolds in the hidden layers.\n\nYou can check the reconstruction error rate. Call `get_reconstruct_error` method as follow.\n\n```python\nreconstruct_error_arr = encoder_decoder_controller.get_reconstruction_error()\n```\n\nIf you want to know how to minimize the reconstructed error, see my Jupyter notebook: [demo/demo_sine_wave_prediction_by_LSTM_encoder_decoder.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_sine_wave_prediction_by_LSTM_encoder_decoder.ipynb).\n\n<a name=\"build_convolutional_auto_encoder\"></a>\n## Usecase: Build Convolutional Auto-Encoder.\n\nSetup logger for verbose output and import Python and Cython modules in the same manner as <a href=\"#usecase_build_encoder_decoder_based_on_LSTM_as_a_reconstruction_model\">Usecase: Build Encoder/Decoder based on LSTM as a reconstruction model</a>.\n\n```python\nfrom logging import getLogger, StreamHandler, NullHandler, DEBUG, ERROR\n\nlogger = getLogger(\"pydbm\")\nhandler = StreamHandler()\nhandler.setLevel(DEBUG)\nlogger.setLevel(DEBUG)\nlogger.addHandler(handler)\n\n# ReLu Function as activation function.\nfrom pydbm.activation.relu_function import ReLuFunction\n# Tanh Function as activation function.\nfrom pydbm.activation.tanh_function import TanhFunction\n# Logistic Function as activation function.\nfrom pydbm.activation.logistic_function import LogisticFunction\n\n# Loss function.\nfrom pydbm.loss.mean_squared_error import MeanSquaredError\n\n# Adam as a optimizer.\nfrom pydbm.optimization.optparams.adam import Adam\n\n# Verification.\nfrom pydbm.verification.verificate_function_approximation import VerificateFunctionApproximation\n```\n\nAnd import Python and Cython modules of the Convolutional Auto-Encoder.\n\n```python\n# Controller of Convolutional Auto-Encoder\nfrom pydbm.cnn.convolutionalneuralnetwork.convolutional_auto_encoder import ConvolutionalAutoEncoder\n# First convolution layer.\nfrom pydbm.cnn.layerablecnn.convolution_layer import ConvolutionLayer as ConvolutionLayer1\n# Second convolution layer.\nfrom pydbm.cnn.layerablecnn.convolution_layer import ConvolutionLayer as ConvolutionLayer2\n# Computation graph for first convolution layer.\nfrom pydbm.synapse.cnn_graph import CNNGraph as ConvGraph1\n# Computation graph for second convolution layer.\nfrom pydbm.synapse.cnn_graph import CNNGraph as ConvGraph2\n```\n\nInstantiate `ConvolutionLayer`s, delegating `CNNGraph`s respectively.\n\n```python\n# First convolution layer.\nconv1 = ConvolutionLayer1(\n    # Computation graph for first convolution layer.\n    ConvGraph1(\n        # Logistic function as activation function.\n        activation_function=LogisticFunction(),\n        # The number of `filter`.\n        filter_num=20,\n        # Channel.\n        channel=1,\n        # The size of kernel.\n        kernel_size=3,\n        # The filter scale.\n        scale=0.1,\n        # The nubmer of stride.\n        stride=1,\n        # The number of zero-padding.\n        pad=1\n    )\n)\n\n# Second convolution layer.\nconv2 = ConvolutionLayer2(\n    # Computation graph for second convolution layer.\n    ConvGraph2(\n        # Computation graph for second convolution layer.\n        activation_function=LogisticFunction(),\n        # The number of `filter`.\n        filter_num=20,\n        # Channel.\n        channel=20,\n        # The size of kernel.\n        kernel_size=3,\n        # The filter scale.\n        scale=0.1,\n        # The nubmer of stride.\n        stride=1,\n        # The number of zero-padding.\n        pad=1\n    )\n)\n```\n\nInstantiate `ConvolutionalAutoEncoder` and setup parameters.\n\n```python\ncnn = ConvolutionalAutoEncoder(\n    # The `list` of `ConvolutionLayer`.\n    layerable_cnn_list=[\n        conv1, \n        conv2\n    ],\n    # The number of epochs in mini-batch training.\n    epochs=200,\n    # The batch size.\n    batch_size=100,\n    # Learning rate.\n    learning_rate=1e-05,\n    # Attenuate the `learning_rate` by a factor of this value every `attenuate_epoch`.\n    learning_attenuate_rate=0.1,\n    # Attenuate the `learning_rate` by a factor of `learning_attenuate_rate` every `attenuate_epoch`.\n    attenuate_epoch=50,\n    # Size of Test data set. If this value is `0`, the validation will not be executed.\n    test_size_rate=0.3,\n    # Optimizer.\n    opt_params=Adam(),\n    # Verification.\n    verificatable_result=VerificateFunctionApproximation(),\n    # The rate of dataset for test.\n    test_size_rate=0.3,\n    # Tolerance for the optimization.\n    # When the loss or score is not improving by at least tol \n    # for two consecutive iterations, convergence is considered \n    # to be reached and training stops.\n    tol=1e-15\n)\n```\n\nExecute learning.\n\n```python\ncnn.learn(img_arr, img_arr)\n```\n\n`img_arr` is a `np.ndarray` of image data, which is a rank-4 array-like or sparse matrix of shape: (`The number of samples`, `Channel`, `Height of image`, `Width of image`), as the first and second argument. If the value of this second argument is not equivalent to the first argument and the shape is (`The number of samples`, `The number of features`), in other words, the rank is 2, the function of `cnn` corresponds to a kind of Regression model.\n\nAfter learning, the `cnn` provides a function of `inference` method.\n\n```python\nresult_arr = cnn.inference(test_img_arr[:100])\n```\n\nThe shape of `test_img_arr` and `result_arr` is equivalent to `img_arr`. \n\nIf you want to know how to visualize the reconstructed images, see my Jupyter notebook: [demo/demo_convolutional_auto_encoder.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_convolutional_auto_encoder.ipynb).\n\n### Save pre-learned parameters.\n\nThe object `cnn`, which is-a `ConvolutionalNeuralNetwork`, has the method `save_pre_learned_params`, to store the pre-learned parameters in compressed <a href=\"https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.lib.format.html#module-numpy.lib.format\" target=\"_blank\">NPY format</a> files.\n\n```python\n# Save pre-learned parameters.\ncnn.save_pre_learned_params(\n    # Path of dir. If `None`, the file is saved in the current directory.\n    dir_path=\"/var/tmp/\",\n    # The naming rule of files. If `None`, this value is `cnn`.\n    file_name=\"demo_cnn\"\n)\n```\n\n### Transfer learning in Convolutional Auto-Encoder.\n\n`__init__` of `ConvolutionalAutoEncoder`, which is-a `ConvolutionalNeuralNetwork`, can be given `pre_learned_path_list` which is a `list` of file paths that store pre-learned parameters.\n\n```python\ncnn2 = ConvolutionalAutoEncoder(\n    layerable_cnn_list=[\n        conv1, \n        conv2\n    ],\n    epochs=100,\n    batch_size=batch_size,\n    learning_rate=1e-05,\n    learning_attenuate_rate=0.1,\n    attenuate_epoch=25,\n    computable_loss=MeanSquaredError(),\n    opt_params=Adam(),\n    verificatable_result=VerificateFunctionApproximation(),\n    test_size_rate=0.3,\n    tol=1e-15,\n    save_flag=True,\n    pre_learned_path_list=[\n        \"pre-learned/demo_cnn_0.npz\",\n        \"pre-learned/demo_cnn_1.npz\"\n    ]\n)\n\n# Execute learning.\ncnn2.learn(img_arr, img_arr)\n```\n\n## Usecase: Build Spatio-Temporal Auto-Encoder.\n\nSetup logger for verbose output and import Python and Cython modules in the same manner as <a href=\"#usecase_build_encoder_decoder_based_on_LSTM_as_a_reconstruction_model\">Usecase: Build Encoder/Decoder based on LSTM as a reconstruction model</a>.\n\nImport Python and Cython modules of the Spatio-Temporal Auto-Encoder.\n\n```python\nfrom pydbm.cnn.spatio_temporal_auto_encoder import SpatioTemporalAutoEncoder\n```\n\nBuild Convolutional Auto-Encoder in the same manner as <a href=\"#build_convolutional_auto_encoder\">Usecase: Build Convolutional Auto-Encoder.</a> and build Encoder/Decoder in the same manner as <a href=\"#usecase_build_encoder_decoder_based_on_LSTM_as_a_reconstruction_model\">Usecase: Build Encoder/Decoder based on LSTM as a reconstruction model</a>.\n\nInstantiate `SpatioTemporalAutoEncoder` and setup parameters.\n\n```python\ncnn = SpatioTemporalAutoEncoder(\n    # The `list` of `LayerableCNN`.\n    layerable_cnn_list=[\n        conv1, \n        conv2\n    ],\n    # is-a `ReconstructableModel`.\n    encoder=encoder,\n    # is-a `ReconstructableModel`.\n    decoder=decoder,\n    # Epochs of Mini-batch.\n    epochs=100,\n    # Batch size of Mini-batch.\n    batch_size=20,\n    # Learning rate.\n    learning_rate=1e-05,\n    # Attenuate the `learning_rate` by a factor of this value every `attenuate_epoch`.\n    learning_attenuate_rate=0.1,\n    # Attenuate the `learning_rate` by a factor of `learning_attenuate_rate` every `attenuate_epoch`.\n    attenuate_epoch=25,\n    # Loss function.\n    computable_loss=MeanSquaredError(),\n    # Optimization function.\n    opt_params=Adam(),\n    # Verification function.\n    verificatable_result=VerificateFunctionApproximation(),\n    # Size of Test data set. If this value is `0`, the validation will not be executed.\n    test_size_rate=0.3,\n    # Tolerance for the optimization.\n    tol=1e-15\n)\n```\n\nExecute learning.\n\n```python\ncnn.learn(img_arr, img_arr)\n```\n\n`img_arr` is a `np.ndarray` of image data, which is a **rank-5** array-like or sparse matrix of shape: (`The number of samples`, `The length of one sequence`, `Channel`, `Height of image`, `Width of image`), as the first and second argument.\n\nAfter learning, the `cnn` provides a function of `inference` method.\n\n```python\nresult_arr = cnn.inference(test_img_arr[:100])\n```\n\nIf you want to know how to visualize the reconstructed video images, see my Jupyter notebook: [demo/demo_spatio_temporal_auto_encoder.ipynb](https://github.com/chimera0/accel-brain-code/blob/master/Deep-Learning-by-means-of-Design-Pattern/demo/demo_spatio_temporal_auto_encoder.ipynb).\n\n### Save pre-learned parameters.\n\nThe object `cnn`, which is-a `SpatioTemporalAutoEncoder`, has the method `save_pre_learned_params`, to store the pre-learned parameters in compressed <a href=\"https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.lib.format.html#module-numpy.lib.format\" target=\"_blank\">NPY format</a> files.\n\n```python\ncnn.save_pre_learned_params(\"/var/tmp/spae/\")\n```\n\n#### Naming rule of saved files.\n\n- `spatio_cnn_X.npz`: Pre-learned parameters in `X` layer of Convolutional Auto-Encoder.\n- `temporal_encoder.npz`: Pre-learned parameters in the Temporal Encoder.\n- `temporal_decoder.npz`: Pre-learned parameters in the Temporal Decoder.\n\n### Transfer learning in Spatio-Temporal Auto-Encoder.\n\n`__init__` method of `SpatioTemporalAutoEncoder` can be given `pre_learned_dir`, which is-a `str` of directory path that stores pre-learned parameters of the Convolutional Auto-Encoder and the Encoder/Decoder based on LSTM.\n\n```python\ncnn2 = SpatioTemporalAutoEncoder(\n    # The `list` of `LayerableCNN`.\n    layerable_cnn_list=[\n        conv1, \n        conv2\n    ],\n    # is-a `ReconstructableModel`.\n    encoder=encoder,\n    # is-a `ReconstructableModel`.\n    decoder=decoder,\n    # Epochs of Mini-batch.\n    epochs=100,\n    # Batch size of Mini-batch.\n    batch_size=20,\n    # Learning rate.\n    learning_rate=1e-05,\n    # Attenuate the `learning_rate` by a factor of this value every `attenuate_epoch`.\n    learning_attenuate_rate=0.1,\n    # Attenuate the `learning_rate` by a factor of `learning_attenuate_rate` every `attenuate_epoch`.\n    attenuate_epoch=25,\n    # Loss function.\n    computable_loss=MeanSquaredError(),\n    # Optimization function.\n    opt_params=Adam(),\n    # Verification function.\n    verificatable_result=VerificateFunctionApproximation(),\n    # Size of Test data set. If this value is `0`, the validation will not be executed.\n    test_size_rate=0.3,\n    # Tolerance for the optimization.\n    tol=1e-15,\n    # Path to directory that stores pre-learned parameters.\n    pre_learned_dir=\"/var/tmp/spae/\"\n)\n\ncnn2.learn(img_arr, img_arr)\n```\n\n## Usecase: Build Optimizer.\n\nIf you want to use various optimizers other than Stochastic Gradient Descent(SGD), instantiate each class as follows.\n\n### Adaptive subgradient methods(AdaGrad).\n\nIf you want to use Adaptive subgradient methods(AdaGrad) optimizer, import `AdaGrad` and instantiate it.\n\n```python\n# AdaGrad as a optimizer.\nfrom pydbm.optimization.optparams.ada_grad import AdaGrad\n\n# is-a `OptParams`.\nopt_params = AdaGrad()\n```\n\n### Adaptive RootMean-Square (RMSProp) gradient decent algorithm.\n\nIf you want to use an optimizer of the Adaptive RootMean-Square (RMSProp) gradient decent algorithm, import `RMSProp` and instantiate it.\n\n```python\n# RMSProp as a optimizer.\nfrom pydbm.optimization.optparams.rms_prop import RMSProp\n\n# is-a `OptParams`.\nopt_params = RMSProp(\n    # Decay rate.\n    decay_rate=0.99\n)\n```\n\n### Nesterov's Accelerated Gradient(NAG).\n\nIf you want to use the Nesterov's Accelerated Gradient(NAG) optimizer, import `NAG` and instantiate it.\n\n```python\n# Adam as a optimizer.\nfrom pydbm.optimization.optparams.nag import NAG\n\n# is-a `OptParams`.\nopt_params = NAG(\n    # Momentum.\n    momentum=0.9\n)\n```\n\n### Adaptive Moment Estimation(Adam).\n\nIf you want to use the Adaptive Moment Estimation(Adam) optimizer, import `Adam` and instantiate it.\n\n```python\n# Adam as a optimizer.\nfrom pydbm.optimization.optparams.adam import Adam\n\n# is-a `OptParams`.\nopt_params = Adam(\n    # BETA 1.\n    beta_1=0.9,\n    # BETA 2.\n    beta_2=0.99,\n    # Compute bias-corrected first moment / second raw moment estimate or not.\n    bias_corrected_flag=False\n)\n```\n\n### Nesterov-accelerated Adaptive Moment Estimation(Nadam).\n\nIf you want to use the Nesterov-accelerated Adaptive Moment Estimation(Nadam) optimizer, import `Nadam` and instantiate it.\n\n```python\n# Nadam as a optimizer.\nfrom pydbm.optimization.optparams.nadam import Nadam\n\n# is-a `OptParams`.\nopt_params = Nadam(\n    # BETA 1.\n    beta_1=0.9,\n    # BETA 2.\n    beta_2=0.99,\n    # Compute bias-corrected first moment / second raw moment estimate or not.\n    bias_corrected_flag=False\n)\n```\n\n## Usecase: Tied-weights.\n\nAn Auto-Encoder is guaranteed to have a well-defined energy function if it has tied weights. It reduces the number of parameters. \n\n<div><blockquote>\n\"It is interesting to note that for an autoencoder whose weights are not tied, contractive regularization will encourage the vector field to be conservative. The reason is that encouraging the first derivative to be small and the second derivative to be negative will tend to bound the energy surface near the training.\"\n</blockquote>\nKamyshanska, H., & Memisevic, R. (2014). The potential energy of an autoencoder. IEEE transactions on pattern analysis and machine intelligence, 37(6), 1261-1273., p7.\n</div>\n\nIn this library, `ConvolutionalAutoEncoder`'s weights are tied in default. But the weight matrixs of `SimpleAutoEncoder` which has two `NeuralNetwork`s are not tied. If you want to tie the weights, set the `tied_graph` as follows.\n\n```python\nfrom pydbm.synapse.nn_graph import NNGraph\nfrom pydbm.activation.identity_function import IdentityFunction\n\n# Encoder's graph.\nencoder_graph = NNGraph(\n    activation_function=IdentityFunction(),\n    hidden_neuron_count=100,\n    output_neuron_count=10,\n)\n\n# Decoder's graph.\ndecoder_graph = NNGraph(\n    activation_function=IdentityFunction(),\n    hidden_neuron_count=10,\n    output_neuron_count=100,\n)\n\n# Set encoder's graph.\ndecoder_graph.tied_graph = encoder_graph\n```\n\n## Usecase: Build and delegate image generator.\n\n`ConvolutionalAutoEncoder` and `SpatioTemporalAutoEncoder`, which are `ConvolutionalNeuralNetwork`s, provide a method `learn_generated` which can be delegated an `ImageGenerator`. `ImageGenerator` is an Iterates to reads batches of images from local directories for mini-batch training.\n\n```python\n# Image generator for Auto-Encoder or Encoder/Decoder scheme.\nfrom pydbm.cnn.featuregenerator.image_generator import ImageGenerator\n\nfeature_generator = ImageGenerator(\n    # Epochs of Mini-batch.\n    epochs=100,\n    # Batch size of Mini-batch.\n    batch_size=20,\n    # Path of directory which stores image files for training.\n    training_image_dir=\"img/training/\",\n    # Path of directory which stores image files for test.\n    test_image_dir=\"img/test/\",\n    # The length of one sequence.\n    # If `None`, generated `np.ndarray` of images will be rank-4 matrices.\n    seq_len=10,\n    # Gray scale or not.\n    gray_scale_flag=True,\n    # Height and width of images. The shape is: Tuple(`width`, `height`).\n    wh_size_tuple=(94, 96),\n    # Normalization mode. `z_score` or `min_max`.\n    norm_mode=\"z_score\"\n)\n```\n\nDelegate `feature_generator` to `cnn`.\n\n```python\ncnn.learn_generated(feature_generator)\n```\n\nMethod `learn_generated` is functionally equivalent to method `learn`.\n\n## Usecase: Save and load the pre-learned parameters in computation graphs.\n\nFor file management of pre-learned parameters in the transfer learning problem setting, each computation graph, which is-a `Synapse`, provides two methods: `save_pre_learned_params` and `load_pre_learned_params`.\n\n```python\n# Save pre-learned parameters.\nsynapse.save_pre_learned_params(\"/var/tmp/pre_learned.npz\")\n```\n\nThe function of this method is to store the pre-learned parameters in compressed <a href=\"https://docs.scipy.org/doc/numpy-1.15.1/reference/generated/numpy.lib.format.html#module-numpy.lib.format\" target=\"_blank\">NPY format</a> files.\n\nAnd call the method `load_pre_learned_params` to execute transfer learning or re-learn.\n\n```python\n# Load pre-learned parameters.\nsynapse.load_pre_learned_params(\"/var/tmp/pre_learned.npz\")\n```\n\nThe class `Synapse` has sub-classes: `CompleteBipartiteGraph`, `LSTMGraph`, `CNNGraph`, and so on. All computation graphs make it possible to do pre-learning and transfer learning.\n\n## References\n\n### Deep Boltzmann machines.\n\n- Ackley, D. H., Hinton, G. E., & Sejnowski, T. J. (1985). A learning algorithm for Boltzmann machines. Cognitive science, 9(1), 147-169.\n- Boulanger-Lewandowski, N., Bengio, Y., & Vincent, P. (2012). Modeling temporal dependencies in high-dimensional sequences: Application to polyphonic music generation and transcription. arXiv preprint arXiv:1206.6392.\n- Eslami, S. A., Heess, N., Williams, C. K., & Winn, J. (2014). The shape boltzmann machine: a strong model of object shape. International Journal of Computer Vision, 107(2), 155-176.\n- Hinton, G. E. (2002). Training products of experts by minimizing contrastive divergence. Neural computation, 14(8), 1771-1800.\n- Le Roux, N., & Bengio, Y. (2008). Representational power of restricted Boltzmann machines and deep belief networks. Neural computation, 20(6), 1631-1649.\n- Lyu, Q., Wu, Z., Zhu, J., & Meng, H. (2015, June). Modelling High-Dimensional Sequences with LSTM-RTRBM: Application to Polyphonic Music Generation. In IJCAI (pp. 4138-4139).\n- Lyu, Q., Wu, Z., & Zhu, J. (2015, October). Polyphonic music modelling with LSTM-RTRBM. In Proceedings of the 23rd ACM international conference on Multimedia (pp. 991-994). ACM.\n- Salakhutdinov, R., & Hinton, G. E. (2009). Deep boltzmann machines. InInternational conference on artificial intelligence and statistics (pp. 448-455).\n- Sutskever, I., Hinton, G. E., & Taylor, G. W. (2009). The recurrent temporal restricted boltzmann machine. In Advances in Neural Information Processing Systems (pp. 1601-1608).\n\n### Encoder/Decoder schemes with an Attention mechanism.\n\n- Bahdanau, D., Cho, K., & Bengio, Y. (2014). Neural machine translation by jointly learning to align and translate. arXiv preprint arXiv:1409.0473.\n- Cho, K., Van Merri\u00ebnboer, B., Gulcehre, C., Bahdanau, D., Bougares, F., Schwenk, H., & Bengio, Y. (2014). Learning phrase representations using RNN encoder-decoder for statistical machine translation. arXiv preprint arXiv:1406.1078.\n- Malhotra, P., Ramakrishnan, A., Anand, G., Vig, L., Agarwal, P., & Shroff, G. (2016). LSTM-based encoder-decoder for multi-sensor anomaly detection. arXiv preprint arXiv:1607.00148.\n- Xingjian, S. H. I., Chen, Z., Wang, H., Yeung, D. Y., Wong, W. K., & Woo, W. C. (2015). Convolutional LSTM network: A machine learning approach for precipitation nowcasting. In Advances in neural information processing systems (pp. 802-810).\n- Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., ... & Polosukhin, I. (2017). Attention is all you need. In Advances in Neural Information Processing Systems (pp. 5998-6008).\n\n### Auto-Encoders.\n\n- Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., & Baskurt, A. (2012, September). Spatio-Temporal Convolutional Sparse Auto-Encoder for Sequence Classification. In BMVC (pp. 1-12).\n- Bengio, Y., Yao, L., Alain, G., & Vincent, P. (2013). Generalized denoising auto-encoders as generative models. In Advances in neural information processing systems (pp. 899-907).\n- Chong, Y. S., & Tay, Y. H. (2017, June). Abnormal event detection in videos using spatiotemporal autoencoder. In International Symposium on Neural Networks (pp. 189-196). Springer, Cham.\n- Masci, J., Meier, U., Cire\u015fan, D., & Schmidhuber, J. (2011, June). Stacked convolutional auto-encoders for hierarchical feature extraction. In International Conference on Artificial Neural Networks (pp. 52-59). Springer, Berlin, Heidelberg.\n- Patraucean, V., Handa, A., & Cipolla, R. (2015). Spatio-temporal video autoencoder with differentiable memory. arXiv preprint arXiv:1511.06309.\n- Rifai, S., Vincent, P., Muller, X., Glorot, X., & Bengio, Y. (2011, June). Contractive auto-encoders: Explicit invariance during feature extraction. In Proceedings of the 28th International Conference on International Conference on Machine Learning (pp. 833-840). Omnipress.\n- Rifai, S., Mesnil, G., Vincent, P., Muller, X., Bengio, Y., Dauphin, Y., & Glorot, X. (2011, September). Higher order contractive auto-encoder. In Joint European Conference on Machine Learning and Knowledge Discovery in Databases (pp. 645-660). Springer, Berlin, Heidelberg.\n- Seung, H. S. (1998). Learning continuous attractors in recurrent networks. In Advances in neural information processing systems (pp. 654-660).\n- Zhao, J., Mathieu, M., & LeCun, Y. (2016). Energy-based generative adversarial network. arXiv preprint arXiv:1609.03126.\n\n### Generative Adversarial Networks(GANs).\n\n- Goodfellow, I., Pouget-Abadie, J., Mirza, M., Xu, B., Warde-Farley, D., Ozair, S., ... & Bengio, Y. (2014). Generative adversarial nets. In Advances in neural information processing systems (pp. 2672-2680).\n- Makhzani, A., Shlens, J., Jaitly, N., Goodfellow, I., & Frey, B. (2015). Adversarial autoencoders. arXiv preprint arXiv:1511.05644.\n- Zhao, J., Mathieu, M., & LeCun, Y. (2016). Energy-based generative adversarial network. arXiv preprint arXiv:1609.03126.\n\n### Unsupervised / Supervised pre-training\n\n- Bengio, Y., Lamblin, P., Popovici, D., & Larochelle, H. (2007). Greedy layer-wise training of deep networks. In Advances in neural information processing systems (pp. 153-160).\n- Erhan, D., Bengio, Y., Courville, A., Manzagol, P. A., Vincent, P., & Bengio, S. (2010). Why does unsupervised pre-training help deep learning?. Journal of Machine Learning Research, 11(Feb), 625-660.\n\n### Representations learning.\n\n- Erhan, D., Courville, A., & Bengio, Y. (2010). Understanding representations learned in deep architectures. Department dInformatique et Recherche Operationnelle, University of Montreal, QC, Canada, Tech. Rep, 1355, 1.\n- Goodfellow, I., Bengio, Y., & Courville, A. (2016). Deep learning (adaptive computation and machine learning series). Adaptive Computation and Machine Learning series, 800.\n\n### Semi-supervised learning.\n\n- Ghifary, M., Kleijn, W. B., Zhang, M., Balduzzi, D., & Li, W. (2016, October). Deep reconstruction-classification networks for unsupervised domain adaptation. In European Conference on Computer Vision (pp. 597-613). Springer, Cham.\n- Rasmus, A., Berglund, M., Honkala, M., Valpola, H., & Raiko, T. (2015). Semi-supervised learning with ladder networks. In Advances in neural information processing systems (pp. 3546-3554).\n- Valpola, H. (2015). From neural PCA to deep unsupervised learning. In Advances in Independent Component Analysis and Learning Machines (pp. 143-171). Academic Press.\n\n### Deep Embedded Clustering.\n\n- Aljalbout, E., Golkov, V., Siddiqui, Y., Strobel, M., & Cremers, D. (2018). Clustering with deep learning: Taxonomy and new methods. arXiv preprint arXiv:1801.07648.\n- Guo, X., Gao, L., Liu, X., & Yin, J. (2017, June). Improved Deep Embedded Clustering with Local Structure Preservation. In IJCAI (pp. 1753-1759).\n- Guo, X., Liu, X., Zhu, E., & Yin, J. (2017, November). Deep clustering with convolutional autoencoders. In International Conference on Neural Information Processing (pp. 373-382). Springer, Cham.\n- Ren, Y., Hu, K., Dai, X., Pan, L., Hoi, S. C., & Xu, Z. (2019). Semi-supervised deep embedded clustering. Neurocomputing, 325, 121-130.\n- Xie, J., Girshick, R., & Farhadi, A. (2016, June). Unsupervised deep embedding for clustering analysis. In International conference on machine learning (pp. 478-487).\n- Zhao, J., Mathieu, M., & LeCun, Y. (2016). Energy-based generative adversarial network. arXiv preprint arXiv:1609.03126.\n- Wagstaff, K., Cardie, C., Rogers, S., & Schr\u00f6dl, S. (2001, June). Constrained k-means clustering with background knowledge. In Icml (Vol. 1, pp. 577-584).\n\n### Optimizations.\n\n- Bengio, Y., Boulanger-Lewandowski, N., & Pascanu, R. (2013, May). Advances in optimizing recurrent networks. In 2013 IEEE International Conference on Acoustics, Speech and Signal Processing (pp. 8624-8628). IEEE.\n- Duchi, J., Hazan, E., & Singer, Y. (2011). Adaptive subgradient methods for online learning and stochastic optimization. Journal of Machine Learning Research, 12(Jul), 2121-2159.\n- Dozat, T. (2016). Incorporating nesterov momentum into adam., Workshop track - ICLR 2016.\n- Kingma, D. P., & Ba, J. (2014). Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980.\n\n### Algorithms, Arithmetic, and Regularizations\n\n- Dumoulin, V., & Visin, F. (2016). A guide to convolution arithmetic for deep learning. arXiv preprint arXiv:1603.07285.\n- He, K., Zhang, X., Ren, S., & Sun, J. (2016). Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition (pp. 770-778).\n- Ioffe, S., & Szegedy, C. (2015). Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03167.\n- Kamyshanska, H., & Memisevic, R. (2014). The potential energy of an autoencoder. IEEE transactions on pattern analysis and machine intelligence, 37(6), 1261-1273.\n- Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., & Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. The Journal of Machine Learning Research, 15(1), 1929-1958.\n- Zaremba, W., Sutskever, I., & Vinyals, O. (2014). Recurrent neural network regularization. arXiv preprint arXiv:1409.2329.\n\n### Related PoC\n\n- [\u91cf\u5b50\u529b\u5b66\u3001\u7d71\u8a08\u529b\u5b66\u3001\u71b1\u529b\u5b66\u306b\u304a\u3051\u308b\u5929\u624d\u7269\u7406\u5b66\u8005\u305f\u3061\u306e\u795e\u5b66\u7684\u306a\u5f62\u8c61\u306b\u3064\u3044\u3066](https://accel-brain.com/das-theologische-bild-genialer-physiker-in-der-quantenmechanik-und-der-statistischen-mechanik-und-thermodynamik/) (Japanese)\n    - [\u71b1\u529b\u5b66\u306e\u524d\u53f2\u3001\u30de\u30af\u30b9\u30a6\u30a7\u30eb\uff1d\u30dc\u30eb\u30c4\u30de\u30f3\u5206\u5e03\u306b\u304a\u3051\u308b\u30a8\u30f3\u30c8\u30ed\u30d4\u30fc\u306e\u6b74\u53f2\u7684\u610f\u5473\u8ad6](https://accel-brain.com/das-theologische-bild-genialer-physiker-in-der-quantenmechanik-und-der-statistischen-mechanik-und-thermodynamik/historische-semantik-der-entropie-in-der-maxwell-boltzmann-verteilung/)\n    - [\u30e1\u30c7\u30a3\u30a2\u3068\u3057\u3066\u306e\u7d71\u8a08\u529b\u5b66\u3068\u5f62\u5f0f\u3068\u3057\u3066\u306e\u30a2\u30f3\u30b5\u30f3\u30d6\u30eb\u3001\u305d\u306e\u30ae\u30d6\u30b9\u7684\u985e\u63a8](https://accel-brain.com/das-theologische-bild-genialer-physiker-in-der-quantenmechanik-und-der-statistischen-mechanik-und-thermodynamik/statistische-mechanik-als-medium-und-ensemble-als-form/)\n    - [\u300c\u30de\u30af\u30b9\u30a6\u30a7\u30eb\u306e\u60aa\u9b54\u300d\u3001\u529b\u5b66\u306e\u57fa\u790e\u6cd5\u5247\u3068\u3057\u3066\u306e\u795e](https://accel-brain.com/das-theologische-bild-genialer-physiker-in-der-quantenmechanik-und-der-statistischen-mechanik-und-thermodynamik/maxwell-damon/)\n- [Web\u30af\u30ed\u30fc\u30e9\u578b\u4eba\u5de5\u77e5\u80fd\u306b\u3088\u308b\u30d1\u30e9\u30c9\u30c3\u30af\u30b9\u63a2\u7d22\u66b4\u9732\u6a5f\u80fd\u306e\u793e\u4f1a\u9032\u5316\u8ad6](https://accel-brain.com/social-evolution-of-exploration-and-exposure-of-paradox-by-web-crawling-type-artificial-intelligence/) (Japanese)\n    - [World-Wide Web\u306e\u793e\u4f1a\u69cb\u9020\u3068Web\u30af\u30ed\u30fc\u30e9\u578b\u4eba\u5de5\u77e5\u80fd\u306e\u610f\u5473\u8ad6](https://accel-brain.com/social-evolution-of-exploration-and-exposure-of-paradox-by-web-crawling-type-artificial-intelligence/sozialstruktur-des-world-wide-web-und-semantik-der-kunstlichen-intelligenz-des-web-crawlers/)\n    - [\u610f\u5473\u8ad6\u306e\u610f\u5473\u8ad6\u3001\u89b3\u5bdf\u306e\u89b3\u5bdf](https://accel-brain.com/social-evolution-of-exploration-and-exposure-of-paradox-by-web-crawling-type-artificial-intelligence/semantik-der-semantik-und-beobachtung-der-beobachtung/)\n- [\u6df1\u5c64\u5f37\u5316\u5b66\u7fd2\u306e\u30d9\u30a4\u30ba\u4e3b\u7fa9\u7684\u306a\u60c5\u5831\u63a2\u7d22\u306b\u99c6\u52d5\u3055\u308c\u305f\u81ea\u7136\u8a00\u8a9e\u51e6\u7406\u306e\u610f\u5473\u8ad6](https://accel-brain.com/semantics-of-natural-language-processing-driven-by-bayesian-information-search-by-deep-reinforcement-learning/) (Japanese)\n    - [\u5e73\u5747\u5834\u8fd1\u4f3c\u63a8\u8ad6\u306e\u7d71\u8a08\u529b\u5b66\u3001\u81ea\u5df1\u7b26\u53f7\u5316\u5668\u3068\u3057\u3066\u306e\u6df1\u5c64\u30dc\u30eb\u30c4\u30de\u30f3\u30de\u30b7\u30f3](https://accel-brain.com/semantics-of-natural-language-processing-driven-by-bayesian-information-search-by-deep-reinforcement-learning/tiefe-boltzmann-maschine-als-selbstkodierer/)\n    - [\u6df1\u5c64\u5b66\u7fd2\u306e\u8a08\u7b97\u30b3\u30b9\u30c8\u524a\u6e1b\u3001MobileNet\u306e\u8a2d\u8a08\u601d\u60f3](https://accel-brain.com/semantics-of-natural-language-processing-driven-by-bayesian-information-search-by-deep-reinforcement-learning/berechnungskostenreduzierung-des-lern-und-designkonzeptes-von-mobilenet/)\n    - [\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u6700\u9069\u5316\u554f\u984c\u306b\u304a\u3051\u308b\u30a2\u30eb\u30b4\u30ea\u30ba\u30e0\u8a2d\u8a08\u306e\u7269\u7406\u5b66\u7684\u306a\u610f\u5473\u8ad6](https://accel-brain.com/semantics-of-natural-language-processing-driven-by-bayesian-information-search-by-deep-reinforcement-learning/physical-semantics-of-algorithm-design-in-neural-network-optimization-problem/)\n    - [\u6b63\u5247\u5316\u554f\u984c\u306b\u304a\u3051\u308b\u6575\u5bfe\u7684\u751f\u6210\u30cd\u30c3\u30c8\u30ef\u30fc\u30af(GANs)\u3068\u6575\u5bfe\u7684\u81ea\u5df1\u7b26\u53f7\u5316\u5668(AAEs)\u306e\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u69cb\u9020](https://accel-brain.com/semantics-of-natural-language-processing-driven-by-bayesian-information-search-by-deep-reinforcement-learning/regularisierungsproblem-und-gan/)\n    - [\u968e\u5c64\u7684\u6f5c\u5728\u5909\u6570\u30e2\u30c7\u30eb\u3092\u30e1\u30c7\u30a3\u30a2\u3068\u3057\u305f\u30e9\u30c0\u30fc\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306e\u534a\u6559\u5e2b\u3042\u308a\u5b66\u7fd2\u5f62\u5f0f\u3001\u30ce\u30a4\u30ba\u9664\u53bb\u578b\u81ea\u5df1\u7b26\u53f7\u5316\u5668\u306e\u6a5f\u80fd](https://accel-brain.com/semantics-of-natural-language-processing-driven-by-bayesian-information-search-by-deep-reinforcement-learning/hierarchical-latent-variable-model-as-media-and-semi-supervised-learning-of-ladder-network-as-a-form/)\n    - [\u30a8\u30cd\u30eb\u30ae\u30fc\u30d9\u30fc\u30b9\u30e2\u30c7\u30eb\u3068\u3057\u3066\u306e\u6575\u5bfe\u7684\u751f\u6210\u30cd\u30c3\u30c8\u30ef\u30fc\u30af(GAN)\u3068\u81ea\u5df1\u7b26\u53f7\u5316\u5668\u306b\u304a\u3051\u308b\u30ea\u30a2\u30d7\u30ce\u30d5\u5b89\u5b9a](https://accel-brain.com/semantics-of-natural-language-processing-driven-by-bayesian-information-search-by-deep-reinforcement-learning/lyaponov-stability-optimization-in-gan-and-auto-encoder-in-energy-based-models/)\n    - [\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u8a00\u8a9e\u30e2\u30c7\u30eb\u306e\u81ea\u7136\u8a00\u8a9e\u51e6\u7406\u3068\u518d\u5e30\u7684\u30cb\u30e5\u30fc\u30e9\u30eb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306e\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u69cb\u9020](https://accel-brain.com/semantics-of-natural-language-processing-driven-by-bayesian-information-search-by-deep-reinforcement-learning/naturliche-sprachverarbeitung-des-neuronalen-netzwerkmodells-und-der-netzwerkstruktur-eines-rekursiven-neuronalen-netzwerks/)\n    - [\u8ee2\u79fb\u5b66\u7fd2\u306e\u30d1\u30e9\u30c9\u30c3\u30af\u30b9\u3001\u8131\u30d1\u30e9\u30c9\u30c3\u30af\u30b9\u5316\u306e\u5f62\u5f0f\u3068\u3057\u3066\u306e\u534a\u6559\u5e2b\u3042\u308a\u5b66\u7fd2](https://accel-brain.com/semantics-of-natural-language-processing-driven-by-bayesian-information-search-by-deep-reinforcement-learning/paradox-of-transfer-learning/)\n- [\u30cf\u30c3\u30ab\u30fc\u502b\u7406\u306b\u6e96\u62e0\u3057\u305f\u4eba\u5de5\u77e5\u80fd\u306e\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u8a2d\u8a08](https://accel-brain.com/architectural-design-of-artificial-intelligence-conforming-to-hacker-ethics/) (Japanese)\n    - [\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u4e2d\u5fc3\u8a2d\u8a08\u306e\u793e\u4f1a\u69cb\u9020\u3068\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u306e\u610f\u5473\u8ad6](https://accel-brain.com/architectural-design-of-artificial-intelligence-conforming-to-hacker-ethics/sozialstruktur-des-architekturzentrum-designs-und-architektur-der-semantik/)\n    - [\u8fd1\u4ee3\u793e\u4f1a\u306e\u793e\u4f1a\u69cb\u9020\u3068\u30cf\u30c3\u30ab\u30fc\u502b\u7406\u306e\u610f\u5473\u8ad6](https://accel-brain.com/architectural-design-of-artificial-intelligence-conforming-to-hacker-ethics/sozialstruktur-der-modernen-gesellschaft-und-semantik-der-hackerethik/)\n    - [\u30e9\u30b7\u30e7\u30ca\u30eb\u7d71\u4e00\u30d7\u30ed\u30bb\u30b9(RUP)\u306e\u793e\u4f1a\u69cb\u9020\u3068\u30a2\u30fc\u30ad\u30c6\u30af\u30c1\u30e3\u8a2d\u8a08\u306e\u610f\u5473\u8ad6](https://accel-brain.com/architectural-design-of-artificial-intelligence-conforming-to-hacker-ethics/sozialstruktur-des-rational-unified-process-und-semantik-des-architekturentwurfs/)\n    - [\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u6307\u5411\u306e\u30aa\u30d6\u30b8\u30a7\u30af\u30c8\u6307\u5411](https://accel-brain.com/architectural-design-of-artificial-intelligence-conforming-to-hacker-ethics/objektorientiert-uber-objektorientiert/)\n- [\u300c\u4eba\u5de5\u306e\u7406\u60f3\u300d\u3092\u80cc\u666f\u3068\u3057\u305f\u300c\u4e07\u7269\u7167\u5fdc\u300d\u306e\u30c7\u30fc\u30bf\u30e2\u30c7\u30ea\u30f3\u30b0](https://accel-brain.com/data-modeling-von-korrespondenz-in-artificial-paradise/) (Japanese)\n    - [\u63a2\u5075\u306e\u6a5f\u80fd\u7684\u7b49\u4fa1\u7269\u3068\u3057\u3066\u306e\u7570\u5e38\u691c\u77e5\u30e2\u30c7\u30eb\u3001\u8b0e\u89e3\u304d\u306e\u63a8\u8ad6\u30a2\u30eb\u30b4\u30ea\u30ba\u30e0](https://accel-brain.com/data-modeling-von-korrespondenz-in-artificial-paradise/anomalieerkennungsmodell-als-funktionelles-aquivalent-eines-detektivs/)\n\n### More detail demos\n\n- [Web\u30af\u30ed\u30fc\u30e9\u578b\u4eba\u5de5\u77e5\u80fd\uff1a\u30ad\u30e1\u30e9\u30fb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306e\u4ed5\u69d8](https://media.accel-brain.com/_chimera-network-is-web-crawling-ai/) (Japanese)\n    - Implemented by the `C++` version of this library, these 20001 bots are able to execute the dimensions reduction(or pre-learning) for natural language processing to run as 20001 web-crawlers and 20001 web-scrapers.\n- [\u30ed\u30dc\u30a2\u30c9\u30d0\u30a4\u30b6\u30fc\u578b\u4eba\u5de5\u77e5\u80fd\uff1a\u30ad\u30e1\u30e9\u30fb\u30cd\u30c3\u30c8\u30ef\u30fc\u30af\u306e\u4ed5\u69d8](https://media.accel-brain.com/_chimera-network-is-robo-adviser/) (Japanese)\n   - The 20001 bots can also simulate the portfolio optimization of securities such as stocks and circulation currency such as cryptocurrencies.\n\n## Author\n\n- chimera0(RUM)\n\n## Author URI\n\n- http://accel-brain.com/\n\n## License\n\n- GNU General Public License v2.0",
        "description_content_type": "text/markdown",
        "docs_url": null,
        "download_url": "",
        "downloads": {
            "last_day": -1,
            "last_month": -1,
            "last_week": -1
        },
        "home_page": "https://github.com/chimera0/accel-brain-code/tree/master/Deep-Learning-by-means-of-Design-Pattern",
        "keywords": "restricted boltzmann machine autoencoder auto-encoder rnn rbm rtrbm convolution deconvolution spatio-temporal encoder decoder LSTM",
        "license": "GPL2",
        "maintainer": "",
        "maintainer_email": "",
        "name": "pydbm",
        "package_url": "https://pypi.org/project/pydbm/",
        "platform": "",
        "project_url": "https://pypi.org/project/pydbm/",
        "project_urls": {
            "Homepage": "https://github.com/chimera0/accel-brain-code/tree/master/Deep-Learning-by-means-of-Design-Pattern"
        },
        "release_url": "https://pypi.org/project/pydbm/1.5.1/",
        "requires_dist": null,
        "requires_python": "",
        "summary": "`pydbm` is Python library for building Restricted Boltzmann Machine(RBM), Deep Boltzmann Machine(DBM), Long Short-Term Memory Recurrent Temporal Restricted Boltzmann Machine(LSTM-RTRBM), and Shape Boltzmann Machine(Shape-BM). From the view points of functionally equivalents and structural expansions, this library also prototypes many variants such as Encoder/Decoder based on LSTM, Convolutional Auto-Encoder, and Spatio-temporal Auto-Encoder.",
        "version": "1.5.1"
    },
    "last_serial": 5867863,
    "releases": {
        "1.0.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "421ef4f5e2daeb942c04470515eacec5",
                    "sha256": "f68c08ceb3358bc1ad078ed585582c3423d807a91c32e6deeb56c0c5bbfe9ee0"
                },
                "downloads": -1,
                "filename": "pydbm-1.0.5.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "421ef4f5e2daeb942c04470515eacec5",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 1906737,
                "upload_time": "2017-10-08T04:12:07",
                "url": "https://files.pythonhosted.org/packages/10/7f/0a0f3c4029f8a4bddc9fe8aa9a6aacbe844405ed035bb29938f228665cc3/pydbm-1.0.5.linux-x86_64.tar.gz"
            }
        ],
        "1.0.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c7494c9791962b80ccac8e163409605b",
                    "sha256": "e8d51910a1a94f9588567f6123992621af22190cc3595b45f7cc58cc03c0c753"
                },
                "downloads": -1,
                "filename": "pydbm-1.0.6.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "c7494c9791962b80ccac8e163409605b",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 1985379,
                "upload_time": "2017-10-08T06:30:57",
                "url": "https://files.pythonhosted.org/packages/a4/92/670d761f1a6f5b44098fea2dd2c0599e426ead3cfdd06e1d91e795a8d9e7/pydbm-1.0.6.linux-x86_64.tar.gz"
            }
        ],
        "1.0.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "35bada2c88ebcec964d366228a827375",
                    "sha256": "fd73043053473013ea47602b59a4f94559e8324b2f04585c89025435d6e8a35b"
                },
                "downloads": -1,
                "filename": "pydbm-1.0.7.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "35bada2c88ebcec964d366228a827375",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 1986560,
                "upload_time": "2017-11-30T12:40:36",
                "url": "https://files.pythonhosted.org/packages/06/64/71451794a7815388c6f3370ba7f5a548447ff35219525ff3c378a3c68b93/pydbm-1.0.7.linux-x86_64.tar.gz"
            }
        ],
        "1.0.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9196fb6ddc76129b9a231e4ce3eb5e14",
                    "sha256": "2d394f98cebef56305993680ab9a21044a3affab38b2d8a61884d5b813e68a7e"
                },
                "downloads": -1,
                "filename": "pydbm-1.0.8.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "9196fb6ddc76129b9a231e4ce3eb5e14",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2026715,
                "upload_time": "2018-02-06T12:13:22",
                "url": "https://files.pythonhosted.org/packages/54/76/dabd70b5e84022a31e5af3f22907b944fc8482e0ab4fd36aed4de61afa68/pydbm-1.0.8.linux-x86_64.tar.gz"
            }
        ],
        "1.0.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8d49ecf4ffda02b2cd518f5f87709913",
                    "sha256": "7bfed0bf50b709db5d396197e7ea5c17193831b27ed09eade7e300a9deb1d7f0"
                },
                "downloads": -1,
                "filename": "pydbm-1.0.9.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "8d49ecf4ffda02b2cd518f5f87709913",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2032452,
                "upload_time": "2018-02-10T11:34:03",
                "url": "https://files.pythonhosted.org/packages/ad/18/56ecfa00a4b560318fbc7f91f04e6ad9c54ab6e060f801b9356edb1b58a9/pydbm-1.0.9.linux-x86_64.tar.gz"
            }
        ],
        "1.1.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c50cea9eae179182d3b042b89c87bb73",
                    "sha256": "3e895a48f4eb4af3c8c531d62c06f75b596ce725a4c5d48276231d73bf3ba87d"
                },
                "downloads": -1,
                "filename": "pydbm-1.1.1.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "c50cea9eae179182d3b042b89c87bb73",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2091565,
                "upload_time": "2018-02-14T12:33:20",
                "url": "https://files.pythonhosted.org/packages/b5/ea/4f283b7d1131e1033c64ed5f71adfeb764332d155087c212f60614697016/pydbm-1.1.1.linux-x86_64.tar.gz"
            }
        ],
        "1.1.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e301f08a550d21c0b55ba8bf60c370ba",
                    "sha256": "22af0ff953a7cf040047f148c024cb58f9d9d8dad03697b448302c29ec3e98f5"
                },
                "downloads": -1,
                "filename": "pydbm-1.1.2.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "e301f08a550d21c0b55ba8bf60c370ba",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2107266,
                "upload_time": "2018-02-21T14:05:33",
                "url": "https://files.pythonhosted.org/packages/be/80/f81de90c506e2961923efae9efa40f6aa01c3913806ea4a4bfb392f924c9/pydbm-1.1.2.linux-x86_64.tar.gz"
            }
        ],
        "1.1.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b2b3003ba0473aa768c4f61f102a4a78",
                    "sha256": "5c372de2b0e2777a9e209ba0bdbcdb705d4023ae4adda296711c944819eb2ef7"
                },
                "downloads": -1,
                "filename": "pydbm-1.1.3.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "b2b3003ba0473aa768c4f61f102a4a78",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2228018,
                "upload_time": "2018-02-22T10:11:47",
                "url": "https://files.pythonhosted.org/packages/49/1b/a3782b9741e012816df2f5b369ffce35d111198fc87566c41500ab16b4c0/pydbm-1.1.3.linux-x86_64.tar.gz"
            }
        ],
        "1.1.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "37e0432ff6ba1c0284adb99888c8ddf3",
                    "sha256": "f5def8a5ae940367ac3d6653ccbfecc1570d78d3231bc70d5afe6f7739937f0a"
                },
                "downloads": -1,
                "filename": "pydbm-1.1.4.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "37e0432ff6ba1c0284adb99888c8ddf3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2234576,
                "upload_time": "2018-02-23T03:05:22",
                "url": "https://files.pythonhosted.org/packages/e4/21/de2f3b266d35fef34bcb7ada9232c7b94463eac6f11b624cb3be487c91cb/pydbm-1.1.4.linux-x86_64.tar.gz"
            }
        ],
        "1.1.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "ff5c0eafbe50ab6649355918a8cb87c3",
                    "sha256": "6a231fd44d82d6859d99b970f8e4cf7cad735b643dc76ae3e99080de165e2f3a"
                },
                "downloads": -1,
                "filename": "pydbm-1.1.5.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "ff5c0eafbe50ab6649355918a8cb87c3",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2305978,
                "upload_time": "2018-02-25T13:15:34",
                "url": "https://files.pythonhosted.org/packages/74/73/d52afeb1b2e108310626e2e6e5bbec19a68afb991b6132b6d17bfcf874c3/pydbm-1.1.5.linux-x86_64.tar.gz"
            }
        ],
        "1.1.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "47ced11314ef4de7ea39f79a4149dce6",
                    "sha256": "c09b321c983749453e65eb7d69dcbad83edf05af0ae0c2b21999d22c9e1e9b46"
                },
                "downloads": -1,
                "filename": "pydbm-1.1.6.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "47ced11314ef4de7ea39f79a4149dce6",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 2365248,
                "upload_time": "2018-03-01T13:49:55",
                "url": "https://files.pythonhosted.org/packages/90/bd/6a2bd80995a43814309a6ba384bbe48b5349c4482a758adf6f5bbbb8bc54/pydbm-1.1.6.linux-x86_64.tar.gz"
            }
        ],
        "1.1.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "542a27b41cdb8fe0b6c1f4c609aaab4c",
                    "sha256": "627d93aa848eab543b1006a359ed634a610782eca41a8d3bb89a724300100868"
                },
                "downloads": -1,
                "filename": "pydbm-1.1.7.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "542a27b41cdb8fe0b6c1f4c609aaab4c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3062970,
                "upload_time": "2018-03-03T02:43:34",
                "url": "https://files.pythonhosted.org/packages/d6/99/08e3543201994a85132549ec82e76da51f28003eb781960e28d764f8d723/pydbm-1.1.7.linux-x86_64.tar.gz"
            }
        ],
        "1.1.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "d7e81f7cc40b3c75ccb50b5d3de8ffff",
                    "sha256": "c3b2972a7b3dcd93cba5181e3b2e12320f8144ac0a635d1eabcf2b14ca11b9be"
                },
                "downloads": -1,
                "filename": "pydbm-1.1.8.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "d7e81f7cc40b3c75ccb50b5d3de8ffff",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3071090,
                "upload_time": "2018-03-03T07:12:27",
                "url": "https://files.pythonhosted.org/packages/75/bf/a3fd6bc548aa7e33ca5647d5d48ca46f0552e0ca3efb44140647fca9a269/pydbm-1.1.8.linux-x86_64.tar.gz"
            }
        ],
        "1.1.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a6dae749393eadb5ab876ed18e96df3a",
                    "sha256": "bff62de2bb9b120fe5f272c2f5c51c02ca81e617bf43a9cf787c9c807ed53396"
                },
                "downloads": -1,
                "filename": "pydbm-1.1.9.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "a6dae749393eadb5ab876ed18e96df3a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3090829,
                "upload_time": "2018-03-11T11:10:11",
                "url": "https://files.pythonhosted.org/packages/4a/57/e0537dc4e4732ecc1aa745c5e5f203628526952b04eb74def7a0d9070a3e/pydbm-1.1.9.linux-x86_64.tar.gz"
            }
        ],
        "1.2.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "33380547bc707e18308a6ec54eb44686",
                    "sha256": "b65d5923aead8098828c15c86e173e7156a9e10b3671eb2c8d58633ab7e24d11"
                },
                "downloads": -1,
                "filename": "pydbm-1.2.0.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "33380547bc707e18308a6ec54eb44686",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3855333,
                "upload_time": "2018-04-29T16:51:01",
                "url": "https://files.pythonhosted.org/packages/32/92/35982cda91bdd95125cc0436d8291f014bcb59fb235586db3bfcb2dc96d9/pydbm-1.2.0.linux-x86_64.tar.gz"
            }
        ],
        "1.2.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "941e6e717e7acb2ca76ae080b46c2a8d",
                    "sha256": "4f073333531f651ca969735434b5a221dc267c1f00f2dec742f3428e8e88672d"
                },
                "downloads": -1,
                "filename": "pydbm-1.2.1.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "941e6e717e7acb2ca76ae080b46c2a8d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3861920,
                "upload_time": "2018-05-06T02:38:20",
                "url": "https://files.pythonhosted.org/packages/51/1a/aba9d779f3435b5c84cb74fef88bdc4b454abe9d73b588728e8c205e9d58/pydbm-1.2.1.linux-x86_64.tar.gz"
            }
        ],
        "1.2.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "fc1cad34792471a1457c787f94a69066",
                    "sha256": "ca2001641af604557f75f5d2f7e695b8bdc6b1f74bcbc5d9806d047c29561ba8"
                },
                "downloads": -1,
                "filename": "pydbm-1.2.2.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "fc1cad34792471a1457c787f94a69066",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3861647,
                "upload_time": "2018-05-06T02:44:06",
                "url": "https://files.pythonhosted.org/packages/24/d8/f65f8c9eec1de05293af7769a32b171fda89ddf2d3ab23e0904398cce0a3/pydbm-1.2.2.linux-x86_64.tar.gz"
            }
        ],
        "1.2.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b606db17c18445c96f8498b6cb10e7c2",
                    "sha256": "eb93577c1a06c89614b0236954ded38aabfecbb933d2fbf90015a2ed651eacf4"
                },
                "downloads": -1,
                "filename": "pydbm-1.2.3.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "b606db17c18445c96f8498b6cb10e7c2",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3296409,
                "upload_time": "2018-05-19T12:34:22",
                "url": "https://files.pythonhosted.org/packages/0e/c1/a5a5ee4da2e85f47ad9a70967b322c2b4344039724cf58a8599896839461/pydbm-1.2.3.linux-x86_64.tar.gz"
            }
        ],
        "1.2.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "780c7fdda87f1f1bf7a349451c69ad16",
                    "sha256": "0d8915aa3fdfdac6514ccf61c82d42aeda983ed42ce91c41f1d741c26870fb65"
                },
                "downloads": -1,
                "filename": "pydbm-1.2.4.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "780c7fdda87f1f1bf7a349451c69ad16",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3318186,
                "upload_time": "2018-06-01T12:35:21",
                "url": "https://files.pythonhosted.org/packages/b5/34/f04ba6fd4e78db6e0ce88ba4956474ea028590fdc3065ea55fbbb10341ce/pydbm-1.2.4.linux-x86_64.tar.gz"
            }
        ],
        "1.2.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "bac7762201cc02ce2f53e13013b03ae2",
                    "sha256": "a45103423c9a5c47247ae404080232fbad2d6cf750bea933ca63fe4253c452c3"
                },
                "downloads": -1,
                "filename": "pydbm-1.2.5.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "bac7762201cc02ce2f53e13013b03ae2",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3431854,
                "upload_time": "2018-06-23T10:35:35",
                "url": "https://files.pythonhosted.org/packages/e3/76/f1769f7d6e6ffcf44d6f859242a7976791bd1834c2ae8dec03852b0a4a1c/pydbm-1.2.5.linux-x86_64.tar.gz"
            }
        ],
        "1.2.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5c4f36635c7ff7f4ad77abb31ce0ed2a",
                    "sha256": "e531452fc2c9d54d550e8e47ed4dfae7d6f24c861df09ac7d74b86cb64346368"
                },
                "downloads": -1,
                "filename": "pydbm-1.2.6.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "5c4f36635c7ff7f4ad77abb31ce0ed2a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 3500431,
                "upload_time": "2018-07-01T09:21:37",
                "url": "https://files.pythonhosted.org/packages/01/a3/cc1b1853df74084d8a3f6c5d070c1515fbfbfcd9b6fc5b3ef22dcea32e71/pydbm-1.2.6.linux-x86_64.tar.gz"
            }
        ],
        "1.2.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "0b13dfe39df2618e39b331c7af45532d",
                    "sha256": "7f5f2c92dd765ee7cee0a593f441927b809b1fbd8e1c43f0f1be8eb0ae7fd920"
                },
                "downloads": -1,
                "filename": "pydbm-1.2.7.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "0b13dfe39df2618e39b331c7af45532d",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5495502,
                "upload_time": "2018-07-14T17:41:08",
                "url": "https://files.pythonhosted.org/packages/20/bf/4c83ee1d32eaaf09cd83c4c844b352ea3f79d68a24c5607b18d879874f8e/pydbm-1.2.7.linux-x86_64.tar.gz"
            }
        ],
        "1.2.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "8187e1eee62191a0654b5f8f1fb14db6",
                    "sha256": "e94e25d2745515c822d7bd11f1fa03c33d1be1faad4600b94aca6df2cf006cb9"
                },
                "downloads": -1,
                "filename": "pydbm-1.2.8.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "8187e1eee62191a0654b5f8f1fb14db6",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5590574,
                "upload_time": "2018-07-15T09:12:04",
                "url": "https://files.pythonhosted.org/packages/13/0c/d9555f88b58c37c28414d64405c6be3885517e72ce9d9c12be7bb99d0376/pydbm-1.2.8.linux-x86_64.tar.gz"
            }
        ],
        "1.2.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "1f486487358aa79572f5250ea8e3936c",
                    "sha256": "c92316cb90b57187b3b6e35cc6dbaefd75a3eaa4609fdbcac1aa12ad0b5a4147"
                },
                "downloads": -1,
                "filename": "pydbm-1.2.9.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "1f486487358aa79572f5250ea8e3936c",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5652412,
                "upload_time": "2018-07-18T12:56:49",
                "url": "https://files.pythonhosted.org/packages/2c/cd/f6f8fcce71eb364af517238275393beb1583437a087e59632034ceaf2560/pydbm-1.2.9.linux-x86_64.tar.gz"
            }
        ],
        "1.3.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "a57a5d4d3508148ec1ae1c10515f3025",
                    "sha256": "3ddbe7f594daa1716c542ac7a9995f083b39d6b0cd70478d44ed46d2e62eacbe"
                },
                "downloads": -1,
                "filename": "pydbm-1.3.0.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "a57a5d4d3508148ec1ae1c10515f3025",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5534749,
                "upload_time": "2018-07-23T14:29:12",
                "url": "https://files.pythonhosted.org/packages/61/72/f7937834f0b557fcbee9bc5cb99f070582e08cc3c54e24f1c46e0b8a48c7/pydbm-1.3.0.linux-x86_64.tar.gz"
            }
        ],
        "1.3.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9f11b6c0c2d6ee56b3cd4e07292d7521",
                    "sha256": "fe77ac97600c1e880ce37adea8614ab2723b1ae151d73dd9db8d720aba740bb8"
                },
                "downloads": -1,
                "filename": "pydbm-1.3.1.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "9f11b6c0c2d6ee56b3cd4e07292d7521",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5599667,
                "upload_time": "2018-07-29T11:52:35",
                "url": "https://files.pythonhosted.org/packages/35/e6/2de1853ed2e1acd90281cfb77177d2b01e300292b79546c9e1bb4159b349/pydbm-1.3.1.linux-x86_64.tar.gz"
            }
        ],
        "1.3.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "9ab6e0f9082e2b88cd833e9935260176",
                    "sha256": "71ffed812bfd4c36a318f77525accd3a7eb6cab9b3a9bc5138a60c85f9e10b69"
                },
                "downloads": -1,
                "filename": "pydbm-1.3.2.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "9ab6e0f9082e2b88cd833e9935260176",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 5789402,
                "upload_time": "2018-08-05T01:47:33",
                "url": "https://files.pythonhosted.org/packages/f8/ea/c3bb6fdf71b85e5191a8f6604a27eefb1cd4c109fb2b268252fca2a62bbc/pydbm-1.3.2.linux-x86_64.tar.gz"
            }
        ],
        "1.3.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "119caee58baf6ae59264c18980f39e75",
                    "sha256": "ea125cf483397d75b1848365e8f09d77919bd7eaf69a2a5f6f91a234f86f2c32"
                },
                "downloads": -1,
                "filename": "pydbm-1.3.3.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "119caee58baf6ae59264c18980f39e75",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 7081295,
                "upload_time": "2018-08-19T11:54:23",
                "url": "https://files.pythonhosted.org/packages/5b/33/4e71cc9c82ba30c5b9685ab9116b9bb84d9a4ef37cb0cee17d41477d003c/pydbm-1.3.3.linux-x86_64.tar.gz"
            }
        ],
        "1.3.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "29462063cd9f518f39ee6f0b1c276a40",
                    "sha256": "0ee842e49fdd26c5d90635a3a97532ecf9844bed7bea164efc7d1b56c8ec6b81"
                },
                "downloads": -1,
                "filename": "pydbm-1.3.4.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "29462063cd9f518f39ee6f0b1c276a40",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 7156967,
                "upload_time": "2018-08-26T10:04:30",
                "url": "https://files.pythonhosted.org/packages/89/8d/3ce9df7de7abb1a0519b7c5a039a0397c9ff86717da77924221c9f9ab9c8/pydbm-1.3.4.linux-x86_64.tar.gz"
            }
        ],
        "1.3.5": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "52cda15e73b1bf0e393ca9eaa3a9a847",
                    "sha256": "7276f1f7cd568d1953febe18e5d57840374d94d048ef7f291346a9a13aad7751"
                },
                "downloads": -1,
                "filename": "pydbm-1.3.5.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "52cda15e73b1bf0e393ca9eaa3a9a847",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 7165260,
                "upload_time": "2018-09-01T10:09:01",
                "url": "https://files.pythonhosted.org/packages/2f/c5/44708e6d6cc00addac72f133f3975f0c343899f466be2881d1e06e221fce/pydbm-1.3.5.linux-x86_64.tar.gz"
            }
        ],
        "1.3.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "c29631ca991a426c93b0f3141b17a584",
                    "sha256": "73f60c954e63140bd44088ecaae50628fcca19893a2b0d0edf1e42f0e7c31c11"
                },
                "downloads": -1,
                "filename": "pydbm-1.3.7.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "c29631ca991a426c93b0f3141b17a584",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8009222,
                "upload_time": "2018-10-09T08:54:09",
                "url": "https://files.pythonhosted.org/packages/3b/05/4f7d14dabda3ebf1f1345e547444aa8377bf5852c24b90de71af05c6741c/pydbm-1.3.7.linux-x86_64.tar.gz"
            }
        ],
        "1.3.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "56e5baa0801bdcbe90f4ea822e14f23a",
                    "sha256": "f46bcdb443905e68bd770fa46bddb1a7614eafca3f4046140e03711a7a541c81"
                },
                "downloads": -1,
                "filename": "pydbm-1.3.8.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "56e5baa0801bdcbe90f4ea822e14f23a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 8609107,
                "upload_time": "2018-11-03T04:19:56",
                "url": "https://files.pythonhosted.org/packages/41/01/cbffa32fde151ade677d977ee7e5ffd6c8eb258284b8240b700a1bb48a74/pydbm-1.3.8.linux-x86_64.tar.gz"
            }
        ],
        "1.3.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "18aff94be25171f0b7350a065b871e98",
                    "sha256": "eab33d68edc1809425bcc951c2cb227a135b0c4e2000415a7421c77a5c55cb65"
                },
                "downloads": -1,
                "filename": "pydbm-1.3.9.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "18aff94be25171f0b7350a065b871e98",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9527210,
                "upload_time": "2018-12-11T15:39:40",
                "url": "https://files.pythonhosted.org/packages/d1/f4/72169552b2199d9e7724043c389ed535cf87d335e7dbb5edc7a84d58e6ff/pydbm-1.3.9.linux-x86_64.tar.gz"
            }
        ],
        "1.4.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "7c34e3df7ec62cdc598b66ae09b498f1",
                    "sha256": "c84125d031cb8570db7abc0d6e1cad63ab9b2b3b54ee1b20a79cb43e479e727d"
                },
                "downloads": -1,
                "filename": "pydbm-1.4.0.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "7c34e3df7ec62cdc598b66ae09b498f1",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9543341,
                "upload_time": "2019-01-25T16:33:28",
                "url": "https://files.pythonhosted.org/packages/d1/b8/0daba073bc566bbc425090e8022587af044a53f030c4960d22b23c6835f2/pydbm-1.4.0.linux-x86_64.tar.gz"
            }
        ],
        "1.4.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e5410dab770c569557a9bbbec2ddd365",
                    "sha256": "456d47215b6d13bbce2ec050cf5dbd18855298eb79098146a8a1375b9e813802"
                },
                "downloads": -1,
                "filename": "pydbm-1.4.1.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "e5410dab770c569557a9bbbec2ddd365",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 9711910,
                "upload_time": "2019-02-17T07:58:35",
                "url": "https://files.pythonhosted.org/packages/e1/fb/18a05b99659cdebb9d9de92f6f91b8b2d04fdfca3ecf73108d740f818d3f/pydbm-1.4.1.linux-x86_64.tar.gz"
            }
        ],
        "1.4.2": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "65a126f08fde6ac9d5a9d5875fbb3e24",
                    "sha256": "6f5a9b89dfb69a20af11d24157cd17b217d8be9399548f13bb9381ef6c84d6e4"
                },
                "downloads": -1,
                "filename": "pydbm-1.4.2.tar.gz",
                "has_sig": false,
                "md5_digest": "65a126f08fde6ac9d5a9d5875fbb3e24",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 14600920,
                "upload_time": "2019-03-02T14:55:07",
                "url": "https://files.pythonhosted.org/packages/47/d7/21c8a730790e9ba308790e2620dd0051d9369f38a9c416c4211031565827/pydbm-1.4.2.tar.gz"
            }
        ],
        "1.4.3": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "5ca4ffd8bbc1fd8206707d71bb128067",
                    "sha256": "77147a2b233013a68e4cce75378ab30daaabe1abfc41e8abeb1fd0af61e1c8f0"
                },
                "downloads": -1,
                "filename": "pydbm-1.4.3.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "5ca4ffd8bbc1fd8206707d71bb128067",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 11368112,
                "upload_time": "2019-03-23T14:45:40",
                "url": "https://files.pythonhosted.org/packages/2e/2b/de7a347afdd7743bf9adf7bc8ddc724fd04a17f9aa133dd1a6c76479b351/pydbm-1.4.3.linux-x86_64.tar.gz"
            }
        ],
        "1.4.4": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "b549b05005654f3a47bd503b4309b398",
                    "sha256": "bb21764287594b8059237bf41713b485bd34fa238c6e94e8782c5a6d14a7373e"
                },
                "downloads": -1,
                "filename": "pydbm-1.4.4.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "b549b05005654f3a47bd503b4309b398",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 12048749,
                "upload_time": "2019-03-30T06:22:26",
                "url": "https://files.pythonhosted.org/packages/d7/2d/535b6daa1418eb4e02ef703ee7b02782ca7483c8080cab6396ba451f9136/pydbm-1.4.4.linux-x86_64.tar.gz"
            }
        ],
        "1.4.6": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "74b5da620aa41f4a6df8e811d8616dd0",
                    "sha256": "15c4be4d58efcf8de78de12b524292f7ff06d921c8331046f27e7dbbd9320249"
                },
                "downloads": -1,
                "filename": "pydbm-1.4.6.linux-x86_64.tar.gz",
                "has_sig": false,
                "md5_digest": "74b5da620aa41f4a6df8e811d8616dd0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 12349666,
                "upload_time": "2019-04-20T15:27:25",
                "url": "https://files.pythonhosted.org/packages/c9/dc/8c832610d23c08271fa887f777fb8a11c26ed6dfefc2498c535101eac037/pydbm-1.4.6.linux-x86_64.tar.gz"
            }
        ],
        "1.4.7": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "98202efb0c493e72ba84b30872c8055a",
                    "sha256": "83e3898b0da6b7f98700014971a00e0f1aee1f16792ad2800ace072f661ee5c1"
                },
                "downloads": -1,
                "filename": "pydbm-1.4.7.tar.gz",
                "has_sig": false,
                "md5_digest": "98202efb0c493e72ba84b30872c8055a",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16057197,
                "upload_time": "2019-06-08T15:30:13",
                "url": "https://files.pythonhosted.org/packages/3d/19/6e5124e89d185d20485ad66ff930ca0a2de9ae7f60b1d7f79d76e78613c3/pydbm-1.4.7.tar.gz"
            }
        ],
        "1.4.8": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "4207ad33adc40695d8a410ec842a9958",
                    "sha256": "210639004b7a40f2f1530a905b36c9db9742671fc1559d61d1fa4baf6ce95b8f"
                },
                "downloads": -1,
                "filename": "pydbm-1.4.8.tar.gz",
                "has_sig": false,
                "md5_digest": "4207ad33adc40695d8a410ec842a9958",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 16847909,
                "upload_time": "2019-06-23T11:27:43",
                "url": "https://files.pythonhosted.org/packages/8c/9f/bdd224e5e0917aed64d0e166401d7868153dfaa84a2b03f5d026430b9e05/pydbm-1.4.8.tar.gz"
            }
        ],
        "1.4.9": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "bb11f2d16bb66ecbd74d6bcfc1eb94bb",
                    "sha256": "7e2d7bb647d2d8f620b1c8f70379564c9fac164ee5a4571a09dcdc61568cff13"
                },
                "downloads": -1,
                "filename": "pydbm-1.4.9.tar.gz",
                "has_sig": false,
                "md5_digest": "bb11f2d16bb66ecbd74d6bcfc1eb94bb",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6470763,
                "upload_time": "2019-07-21T14:10:35",
                "url": "https://files.pythonhosted.org/packages/04/cb/c66a8492f14acda7eaae76d1e59dae16bc04c06d4065d9841547efda0061/pydbm-1.4.9.tar.gz"
            }
        ],
        "1.5.0": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "156a525bb2107cbd27e05a3e4e0c0d61",
                    "sha256": "0d8b30a6cd43970e425c0658855a92bd9103a86fcb4c8a3c010b096ae65f02ad"
                },
                "downloads": -1,
                "filename": "pydbm-1.5.0.tar.gz",
                "has_sig": false,
                "md5_digest": "156a525bb2107cbd27e05a3e4e0c0d61",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6613248,
                "upload_time": "2019-08-04T03:56:24",
                "url": "https://files.pythonhosted.org/packages/76/54/a420a45fb2c36b59b0992a6432a25cc1c2162d311163458cfb61ddda7796/pydbm-1.5.0.tar.gz"
            }
        ],
        "1.5.1": [
            {
                "comment_text": "",
                "digests": {
                    "md5": "e5469d0923e8c4e5b3626669a2a67aa0",
                    "sha256": "693fa25ec2fd35e731feaea5faec4794c23d2f06fb4a4d40742448a527d8b48f"
                },
                "downloads": -1,
                "filename": "pydbm-1.5.1.tar.gz",
                "has_sig": false,
                "md5_digest": "e5469d0923e8c4e5b3626669a2a67aa0",
                "packagetype": "sdist",
                "python_version": "source",
                "requires_python": null,
                "size": 6929859,
                "upload_time": "2019-09-22T03:56:48",
                "url": "https://files.pythonhosted.org/packages/78/cf/a19f4c62c134acb9b2660b92a9ebd35835360aae42fcc606f50cb10d9abc/pydbm-1.5.1.tar.gz"
            }
        ]
    },
    "urls": [
        {
            "comment_text": "",
            "digests": {
                "md5": "e5469d0923e8c4e5b3626669a2a67aa0",
                "sha256": "693fa25ec2fd35e731feaea5faec4794c23d2f06fb4a4d40742448a527d8b48f"
            },
            "downloads": -1,
            "filename": "pydbm-1.5.1.tar.gz",
            "has_sig": false,
            "md5_digest": "e5469d0923e8c4e5b3626669a2a67aa0",
            "packagetype": "sdist",
            "python_version": "source",
            "requires_python": null,
            "size": 6929859,
            "upload_time": "2019-09-22T03:56:48",
            "url": "https://files.pythonhosted.org/packages/78/cf/a19f4c62c134acb9b2660b92a9ebd35835360aae42fcc606f50cb10d9abc/pydbm-1.5.1.tar.gz"
        }
    ]
}