MXNet - Python API

MXNet - Python API

>>> import mxnet as mx

NDArray API

Overview

This document lists the routines of the n-dimensional array package

类	说明
mxnet.ndarray	NDArray API of MXNet.

A NDArray is a multidimensional container of items of the same type and size. Various methods for data manipulation and computation are provided.

>>> x = mx.nd.array([[1, 2, 3], [4, 5, 6]])
>>> type(x)
'mxnet.ndarray.NDArray'>
>>> x.shape
(2, 3)
>>> y = x + mx.nd.ones(x.shape)*3
>>> print(y.asnumpy())
[[ 4.  5.  6.]
 [ 7.  8.  9.]]
>>> z = y.as_in_context(mx.gpu(0))
>>> print(z)
2x3 @gpu(0)>

A detailed tutorial is available at http://mxnet.io/tutorials/python/ndarray.html..

Note

mxnet.ndarray is similar to numpy.ndarray in some aspects. But the difference is not negligible. For example

• NDArray.T does real data transpose to return new a copied array, instead of returning a view of the input array.
• ndarray.dot performs dot between the last axis of the first input array and the first axis of the second input, while numpy.dot uses the second last axis of the input array.

In additional, NDArray supports GPU computation and various neural network layers.

Note

ndarray also provides almost same routines to symbol. Most routines between these two packages share the same C++ operator source codes. But ndarray differs to symbol in several aspects:

• ndarray adopts imperative programming, namely sentences are executed step-by-step so that the results can be obtained immediately.
• Most binary operators such as + and > are enabled broadcasting in default.

In the rest of this document, we first overview the methods provided by the ndarray.NDArray class, and then list other routines provided by the ndarray package.

The NDArray class

Array attributes

属性	说明
NDArray.shape	Tuple of array dimensions.
NDArray.size	Number of elements in the array.
NDArray.context	Device context of the array.
NDArray.dtype	Data-type of the array’s elements.

Array conversion

变换	说明
NDArray.copy	Makes a copy of this NDArray, keeping the same context.
NDArray.copyto	Copies the value of this array to another array.
NDArray.as_in_context	Returns an array on the target device with the same value as this array.
NDArray.asnumpy	Returns a numpy.ndarray object with value copied from this array.
NDArray.asscalar	Returns a scalar whose value is copied from this array.
NDArray.astype	Returns a copy of the array after casting to a specified type.

Array change shape

改变维数	说明
NDArray.T	Returns a copy of the array with axes transposed.
NDArray.reshape	Returns a view of this array with a new shape without altering any data.
NDArray.broadcast_to	Broadcasts the input array to a new shape.

Arithmetic operations

算术操作	说明
NDArray.__add__	x.__add__(y) <=> x+y <=> mx.nd.add(x, y)
NDArray.__sub__	x.__sub__(y) <=> x-y <=> mx.nd.subtract(x, y)
NDArray.__rsub__	x.__rsub__(y) <=> y-x <=> mx.nd.subtract(y, x)
NDArray.__neg__	x.__neg__(y) <=> -x
NDArray.__mul__	x.__mul__(y) <=> x*y <=> mx.nd.multiply(x, y)
NDArray.__div__	x.__div__(y) <=> x/y <=> mx.nd.divide(x, y)
NDArray.__rdiv__	x.__rdiv__(y) <=> y/x <=> mx.nd.divide(y, x)
NDArray.__pow__	x.__pow__(y) <=> x**y <=> mx.nd.power(x,y)

In-place arithmetic operations

本地算术操作	说明
NDArray.__iadd__	x.__iadd__(y) <=> x+=y
NDArray.__isub__	x.__isub__(y) <=> x-=y
NDArray.__imul__	x.__imul__(y) <=> x*=y
NDArray.__idiv__	x.__rdiv__(y) <=> x/=y

Comparison operators

比较操作	说明
NDArray.__lt__	x.__lt__(y) <=> x mx.nd.lesser(x, y)
NDArray.__le__	x.__le__(y) <=> x<=y <=> mx.nd.less_equal(x, y)
NDArray.__gt__	x.__gt__(y) <=> x>y <=> mx.nd.greater(x, y)
NDArray.__ge__	x.__ge__(y) <=> x>=y <=> mx.nd.greater_equal(x, y)
NDArray.__eq__	x.__eq__(y) <=> x==y <=> mx.nd.equal(x, y)
NDArray.__ne__	x.__ne__(y) <=> x!=y <=> mx.nd.not_equal(x, y)

Indexing

索引	说明
NDArray.__getitem__	x.__getitem__(i) <=> x[i]
NDArray.__setitem__	x.__setitem__(i, y) <=> x[i]=y

Lazy evaluation

懒惰（??）	说明
NDArray.wait_to_read	Waits until all previous write operations on the current array are finished.

Array creation routines

创建程序	说明
array	Creates an array from any object exposing the array interface.
empty	Returns a new array of given shape and type, without initializing entries.
zeros	Returns a new array filled with all zeros, with the given shape and type.
ones	Returns a new array filled with all ones, with the given shape and type.
full	Returns a new array of given shape and type, filled with the given value val.
arange	Returns evenly spaced values within a given interval.
load	Loads an array from file.
save	Saves a list of arrays or a dict of str->array to file.

Array manipulation routines

Changing array shape and type

更改类型与形状	说明
cast	Casts all elements of the input to a new type.
reshape	Reshapes the input array.
flatten	Flattens the input array into a 2-D array by collapsing the higher dimensions.
expand_dims	Inserts a new axis of size 1 into the array shape

Expanding array elements

扩展元素	说明
broadcast_to	Broadcasts the input array to a new shape.
broadcast_axes	Broadcasts the input array over particular axes.
repeat	Repeats elements of an array.
tile	Repeats the whole array multiple times.
pad	Pads an array.

Rearranging elements

元素重排	说明
transpose	Permutes the dimensions of an array.
swapaxes	Interchanges two axes of an array.
flip	Reverses the order of elements along given axis while preserving array shape.

Joining and splitting arrays

连接与切片	说明
concat	Joins input arrays along a given axis.
split	Splits an array along a particular axis into multiple sub-arrays.

Indexing routines

索引程序	说明
slice	Slices a contiguous region of the array.
slice_axis	Slices along a given axis.
take	Takes elements from an input array along the given axis.
batch_take	Takes elements from a data batch.
one_hot	Returns a one-hot array.
pick	Picks elements from an input array according to the input indices along the given axis.

Mathematical functions

Arithmetic operations

算术操作	说明
add	Returns element-wise sum of the input arrays with broadcasting.
subtract	Returns element-wise difference of the input arrays with broadcasting.
negative	Numerical negative, element-wise.
multiply	Returns element-wise product of the input arrays with broadcasting.
divide	Returns element-wise division of the input arrays with broadcasting.
dot	Dot product of two arrays.
batch_dot	Batchwise dot product.
add_n	Adds all input arguments element-wise.

Trigonometric functions

三角函数	说明
sin	Computes the element-wise sine of the input array.
cos	Computes the element-wise cosine of the input array.
tan	Computes the element-wise tangent of the input array.
arcsin	Returns element-wise inverse sine of the input array.
arccos	Returns element-wise inverse cosine of the input array.
arctan	Returns element-wise inverse tangent of the input array.
degrees	Converts each element of the input array from radians to degrees.
radians	Converts each element of the input array from degrees to radians.

Hyperbolic functions

双曲函数	说明
sinh	Returns the hyperbolic sine of the input array, computed element-wise.
cosh	Returns the hyperbolic cosine of the input array, computed element-wise.
tanh	Returns the hyperbolic tangent of the input array, computed element-wise.
arcsinh	Returns the element-wise inverse hyperbolic sine of the input array, computed element-wise.
arccosh	Returns the element-wise inverse hyperbolic cosine of the input array, computed element-wise.
arctanh	Returns the element-wise inverse hyperbolic tangent of the input array, computed element-wise.

Reduce functions

减少函数	说明
sum	Computes the sum of array elements over given axes.
nansum	Computes the sum of array elements over given axes treating Not a Numbers (NaN) as zero.
prod	Computes the product of array elements over given axes.
nanprod	Computes the product of array elements over given axes treating Not a Numbers (NaN) as one.
mean	Computes the mean of array elements over given axes.
max	Computes the max of array elements over given axes.
min	Computes the min of array elements over given axes.
norm	Flattens the input array and then computes the l2 norm.

Rounding

四舍五入	说明
round	Returns element-wise rounded value to the nearest integer of the input.
rint	Returns element-wise rounded value to the nearest integer of the input.
fix	Returns element-wise rounded value to the nearest integer towards zero of the input.
floor	Returns element-wise floor of the input.
ceil	Returns element-wise ceiling of the input.

Exponents and logarithms

指数与对数	说明
exp	Returns element-wise exponential value of the input.
expm1	Returns exp(x) - 1 computed element-wise on the input.
log	Returns element-wise Natural logarithmic value of the input.
log10	Returns element-wise Base-10 logarithmic value of the input.
log2	Returns element-wise Base-2 logarithmic value of the input.
log1p	Returns element-wise log(1 + x) value of the input.

Powers

幂	说明
power	Returns result of first array elements raised to powers from second array, element-wise with broadcasting.
sqrt	Returns element-wise square-root value of the input.
rsqrt	Returns element-wise inverse square-root value of the input.
square	Returns element-wise squared value of the input.

Logic functions

逻辑函数	说明
equal	Returns the result of element-wise equal to (==) comparison operation with broadcasting.
not_equal	Returns the result of element-wise not equal to (!=) comparison operation with broadcasting.
greater	Returns the result of element-wise greater than (>) comparison operation with broadcasting.
greater_equal	Returns the result of element-wise greater than or equal to (>=) comparison operation with broadcasting.
lesser	Returns the result of element-wise lesser than (<) comparison operation with broadcasting.
lesser_equal	Returns the result of element-wise lesser than or equal to (<=) comparison operation with broadcasting.

Random sampling

随机样例	说明
uniform	Draw samples from a uniform distribution.
normal	Draw random samples from a normal (Gaussian) distribution.
mxnet.random.seed	Seeds the random number generators in MXNet.

Sorting and searching

排序与搜索	说明
sort	Returns a sorted copy of an input array along the given axis.
topk	Returns the top k elements in an input array along the given axis.
argsort	Returns the indices that would sort an input array along the given axis.
argmax	Returns indices of the maximum values along an axis.
argmin	Returns indices of the minimum values along an axis.

Miscellaneous

杂项	说明
maximum	Returns element-wise maximum of the input arrays with broadcasting.
minimum	Returns element-wise minimum of the input arrays with broadcasting.
clip	Clips (limits) the values in an array.
abs	Returns element-wise absolute value of the input.
sign	Returns element-wise sign of the input.
gamma	Returns the gamma function (extension of the factorial function to the reals) , computed element-wise on the input array.
gammaln	Returns element-wise log of the absolute value of the gamma function of the input.

Neural network

Basic

基本	说明
FullyConnected	Applies a linear transformation: Y=XWT+bY=XWT+b .
Convolution	Compute N-D convolution on (N+2)-D input.
Activation	Applies an activation function element-wise to the input.
BatchNorm	Batch normalization.
Pooling	Performs pooling on the input.
SoftmaxOutput	Computes softmax with logit loss.
softmax	Applies the softmax function.
log_softmax	Computes the log softmax of the input.

More

更多	说明
Correlation	Applies correlation to inputs.
Deconvolution	Applies deconvolution to input and adds a bias.
RNN	Applies a recurrent layer to input.
Embedding	Maps integer indices to vector representations (embeddings).
LeakyReLU	Applies leaky ReLU activation element-wise to the input.
InstanceNorm	An operator taking in a n-dimensional input tensor (n > 2), and normalizing the input by subtracting the mean and variance calculated over the spatial dimensions.
L2Normalization	Normalize the input array using the L2 norm.
LRN	Applies convolution to input and then adds a bias.
ROIPooling	Performs region of interest(ROI) pooling on the input array.
SoftmaxActivation	Applies softmax activation to input.
Dropout	Applies dropout to input.
BilinearSampler	Applies bilinear sampling to input feature map, which is the key of “[NIPS2015] Spatial Transformer Networks” output[batch, channel, y_dst, x_dst] = G(data[batch, channel, y_src, x_src) x_dst, y_dst enumerate all spatial locations in output x_src = grid[batch, 0, y_dst, x_dst] y_src = grid[batch, 1, y_dst, x_dst] G() denotes the bilinear interpolation kernel The out-boundary points will be padded as zeros.
GridGenerator	Generates sampling grid for bilinear sampling.
UpSampling	Performs nearest neighbor/bilinear up sampling to inputs
SpatialTransformer	Applies a spatial transformer to input feature map.
LinearRegressionOutput	Computes and optimizes for squared loss.
LogisticRegressionOutput	Applies a logistic function to the input.
MAERegressionOutput	Computes mean absolute error of the input.
SVMOutput	Computes support vector machine based transformation of the input.
softmax_cross_entropy	Calculate cross_entropy(data, one_hot(label))
smooth_l1	Calculate Smooth L1 Loss(lhs, scalar)
IdentityAttachKLSparseReg	Apply a sparse regularization to the output a sigmoid activation function.
MakeLoss	Make your own loss function in network construction.
BlockGrad	Stops gradient computation.
Custom	Apply a custom operator implemented in a frontend language (like Python).

Symbol API

Overview

This document lists the routines of the symbolic expression package:

类	说明
mxnet.symbol	Symbolic configuration API of MXNet.

A symbol declares computation. It is composited by operators, such as simple matrix operations (e.g. “+”), or a neural network layer (e.g. convolution layer). We can bind data to a symbol to execute the computation.

>>> a = mx.sym.Variable('a')
>>> b = mx.sym.Variable('b')
>>> c = 2 * a + b
>>> type(c)
<class 'mxnet.symbol.Symbol'>
>>> e = c.bind(mx.cpu(), {'a': mx.nd.array([1,2]), 'b':mx.nd.array([2,3])})
>>> y = e.forward()
>>> y
[]
>>> y[0].asnumpy()
array([ 4.,  7.], dtype=float32)

A detailed tutorial is available at http://mxnet.io/tutorials/python/symbol.html.

Note

most operators provided in symbol are similar to ndarray. But also note that symbol differs to ndarray in several aspects:

• symbol adopts declare programming. In other words, we need to first composite the computations, and then feed with data to execute.
• Most binary operators such as + and > are not enabled broadcasting. We need to call the broadcasted version such as broadcast_plus explicitly.

In the rest of this document, we first overview the methods provided by the symbol.Symbol class, and then list other routines provided by the symbol package.

The Symbol class

Composition

Composite multiple symbols into a new one by an operator.

组成	说明
Symbol.__call__	Compose symbol on inputs.

Arithmetic operations

算术操作	说明
Symbol.__add__	x.__add__(y) <=> x+y
Symbol.__sub__	x.__sub__(y) <=> x-y
Symbol.__rsub__	x.__rsub__(y) <=> y-x
Symbol.__neg__	x.__neg__(y) <=> -x
Symbol.__mul__	x.__mul__(y) <=> x*y
Symbol.__div__	x.__div__(y) <=> x/y
Symbol.__rdiv__	x.__rdiv__(y) <=> y/x
Symbol.__pow__	x.__pow__(y) <=> x**y

Comparison operators

比较操作	说明
Symbol.__lt__	x.__lt__(y) <=> x
Symbol.__le__	x.__le__(y) <=> x<=y
Symbol.__gt__	x.__gt__(y) <=> x>y
Symbol.__ge__	x.__ge__(y) <=> x>=y
Symbol.__eq__	x.__eq__(y) <=> x==y
Symbol.__ne__	x.__ne__(y) <=> x!=y

Query information

查询信息	说明
Symbol.name	Get name string from the symbol, this function only works for non-grouped symbol.
Symbol.list_arguments	Lists all the arguments in the symbol.
Symbol.list_outputs	Lists all the outputs in the symbol.
Symbol.list_auxiliary_states	Lists all the auxiliary states in the symbol.
Symbol.list_attr	Gets all attributes from the symbol.
Symbol.attr	Gets attribute string from the symbol.
Symbol.attr_dict	Recursively gets all attributes from the symbol and its children.

Get internal and output symbol

获取内部和输出符号	说明
Symbol.__getitem__	x.__getitem__(i) <=> x[i]
Symbol.__iter__	Returns all outputs in a list
Symbol.get_internals	Gets a new grouped symbol sgroup.
Symbol.get_children	Gets a new grouped symbol whose output contains inputs to output nodes of the original symbol.

Inference type and shape

推理类型与形状	说明
Symbol.infer_type	Infers the type of all arguments and all outputs, given the known types for some arguments.
Symbol.infer_shape	Infers the shapes of all arguments and all outputs given the known shapes of some arguments.
Symbol.infer_shape_partial	Infers the shape partially.

Bind

绑定	说明
Symbol.bind	Bind current symbol to get an executor.
Symbol.simple_bind	Bind current symbol to get an executor, allocate all the ndarrays needed.

Save

保存	说明
Symbol.save	Saves symbol to a file.
Symbol.tojson	Saves symbol to a JSON string.
Symbol.debug_str	Gets a debug string.

Symbol creation routines

符号创建程序	说明
var	Create a symbolic variable with specified name.
zeros	Return a new symbol of given shape and type, filled with zeros.
ones	Return a new symbol of given shape and type, filled with ones.
arange	Return evenly spaced values within a given interval.

Symbol manipulation routines

Changing shape and type

改变形状与类型	说明
cast	Casts all elements of the input to the new type.
reshape	Reshapes the input array into a new shape.
flatten	Flattens the input array into a 2-D array by collapsing the higher dimensions.
expand_dims	Insert a new axis with size 1 into the array shape

Expanding elements

扩展元素	说明
broadcast_to	Broadcasts the input array to a new shape.
broadcast_axes	Broadcasts the input array over particular axes.
repeat	Repeat elements of an array.
tile	Repeat the whole array by multiple times.
pad	Pad an array.

Rearranging elements

元素重排	说明
transpose	Permute the dimensions of an array.
swapaxes	Interchange two axes of an array.
flip	Reverse elements of an array with axis

Joining and splitting symbols

符号式连接与切片	说明
concat	Join input arrays along the given axis.
split	Split an array along a particular axis into multiple sub-arrays.

Indexing routines

索引程序	说明
slice	Slice a continuous region of the array.
slice_axis	Slice along a given axis.
take	Takes elements from an input array along the given axis.
batch_take	Takes elements from a data batch.
one_hot	Returns a one-hot array.

Mathematical functions

Arithmetic operations

算术操作	说明
broadcast_add	Returns element-wise sum of the input arrays with broadcasting.
broadcast_sub	Returns element-wise difference of the input arrays with broadcasting.
broadcast_mul	Returns element-wise product of the input arrays with broadcasting.
broadcast_div	Returns element-wise division of the input arrays with broadcasting.
negative	Negate src
dot	Dot product of two arrays.
batch_dot	Batchwise dot product.
add_n	Add all input arguments element-wise.

Trigonometric functions

三角函数	说明
sin	Computes the element-wise sine of the input.
cos	Computes the element-wise cosine of the input array.
tan	Computes the element-wise tangent of the input array.
arcsin	Returns element-wise inverse sine of the input array.
arccos	Returns element-wise inverse cosine of the input array.
arctan	Returns element-wise inverse tangent of the input array.
hypot	minimum left and right
broadcast_hypot	Returns the hypotenuse of a right angled triangle, given its “legs” with broadcasting.
degrees	Converts each element of the input array from radians to degrees.
radians	Converts each element of the input array from degrees to radians.

Hyperbolic functions

双曲函数	说明
sinh	Returns the hyperbolic sine of the input array, computed element-wise.
cosh	Returns the hyperbolic cosine of the input array, computed element-wise.
tanh	Returns the hyperbolic tangent of the input array, computed element-wise.
arcsinh	Returns the element-wise inverse hyperbolic sine of the input array, computed element-wise.
arccosh	Returns the element-wise inverse hyperbolic cosine of the input array, computed element-wise.
arctanh	Returns the element-wise inverse hyperbolic tangent of the input array, computed element-wise.

Reduce functions

减少函数	说明
sum	Compute the sum of array elements over given axes.
nansum	Compute the sum of array elements over given axes with NaN ignored
prod	Compute the product of array elements over given axes.
nanprod	Compute the product of array elements over given axes with NaN ignored
mean	Compute the mean of array elements over given axes.
max	Compute the max of array elements over given axes.
min	Compute the min of array elements over given axes.
norm	Computes the L2 norm of the input array.

Rounding

四舍五入	说明
round	Returns element-wise rounded value to the nearest integer of the input.
rint	Returns element-wise rounded value to the nearest integer of the input.
fix	Returns element-wise rounded value to the nearest integer towards zero of the input.
floor	Returns element-wise floor of the input.
ceil	Returns element-wise ceiling of the input.

Exponents and logarithms

指数与对数	说明
exp	Returns element-wise exponential value of the input.
expm1	Returns exp(x) - 1 computed element-wise on the input.
log	Returns element-wise Natural logarithmic value of the input.
log10	Returns element-wise Base-10 logarithmic value of the input.
log2	Returns element-wise Base-2 logarithmic value of the input.
log1p	Returns element-wise log(1 + x) value of the input.

Powers

幂	说明
broadcast_power	Returns result of first array elements raised to powers from second array, element-wise with broadcasting.
sqrt	Returns element-wise square-root value of the input.
rsqrt	Returns element-wise inverse square-root value of the input.
square	Returns element-wise squared value of the input.

Logic functions

逻辑函数	说明
broadcast_equal	Returns the result of element-wise equal to (==) comparison operation with broadcasting.
broadcast_not_equal	Returns the result of element-wise not equal to (!=) comparison operation with broadcasting.
broadcast_greater	Returns the result of element-wise greater than (>) comparison operation with broadcasting.
broadcast_greater_equal	Returns the result of element-wise greater than or equal to (>=) comparison operation with broadcasting.
broadcast_lesser	Returns the result of element-wise lesser than (<) comparison operation with broadcasting.
broadcast_lesser_equal	Returns the result of element-wise lesser than or equal to (<=) comparison operation with broadcasting.

Random sampling

随机抽样	说明
uniform	Draw samples from a uniform distribution.
normal	Draw random samples from a normal (Gaussian) distribution.
mxnet.random.seed	Seed the random number generators in MXNet.

Sorting and searching

排序与搜索	说明
sort	Returns a sorted copy of an input array along the given axis.
topk	Returns the top k elements in an input array along the given axis.
argsort	Returns the indices that would sort an input array along the given axis.
argmax	Returns indices of the maximum values along an axis.
argmin	Returns indices of the minimum values along an axis.

Miscellaneous

杂项	说明
maximum	maximum left and right
minimum	minimum left and right
broadcast_maximum	Returns element-wise maximum of the input arrays with broadcasting.
broadcast_minimum	Returns element-wise minimum of the input arrays with broadcasting.
clip	Clip (limit) the values in an array.
abs	Returns element-wise absolute value of the input.
sign	Returns element-wise sign of the input.
gamma	Returns the gamma function (extension of the factorial function to the reals) , computed element-wise on the input array.
gammaln	Returns element-wise log of the absolute value of the gamma function of the input.

Neural network

Basic

基本	说明
FullyConnected	Apply a linear transformation: Y=XWT+bY=XWT+b .
Convolution	Compute N-D convolution on (N+2)-D input.
Activation	Elementwise activation function.
BatchNorm	Batch normalization.
Pooling	Perform pooling on the input.
SoftmaxOutput	Softmax with logit loss.
softmax	Applies the softmax function.
log_softmax	Compute the log softmax of the input.

More

更多	说明
Correlation	Apply correlation to inputs
Deconvolution	Apply deconvolution to input then add a bias.
RNN	Apply a recurrent layer to input.
Embedding	Maps integer indices to vector representations (embeddings).
LeakyReLU	Leaky ReLu activation
InstanceNorm	An operator taking in a n-dimensional input tensor (n > 2), and normalizing the input by subtracting the mean and variance calculated over the spatial dimensions.
L2Normalization	Set the l2 norm of each instance to a constant.
LRN	Apply convolution to input then add a bias.
ROIPooling	Performs region of interest(ROI) pooling on the input array.
SoftmaxActivation	Apply softmax activation to input.
Dropout	Apply dropout to input.
BilinearSampler	Apply bilinear sampling to input feature map, which is the key of “[NIPS2015] Spatial Transformer Networks” output[batch, channel, y_dst, x_dst] = G(data[batch, channel, y_src, x_src) x_dst, y_dst enumerate all spatial locations in output x_src = grid[batch, 0, y_dst, x_dst] y_src = grid[batch, 1, y_dst, x_dst] G() denotes the bilinear interpolation kernel The out-boundary points will be padded as zeros.
GridGenerator	generate sampling grid for bilinear sampling.
UpSampling	Perform nearest neighboor/bilinear up sampling to inputs This function support variable length of positional input.
SpatialTransformer	Apply spatial transformer to input feature map.
LinearRegressionOutput	LinearRegressionOutput computes and optimizes for squared loss.
LogisticRegressionOutput	LogisticRegressionOutput applies a logistic function to the input.
MAERegressionOutput	MAERegressionOutput function computes mean absolute error.
SVMOutput	Computes support vector machine based transformation of the input.
softmax_cross_entropy	Calculate cross_entropy(data, one_hot(label))
smooth_l1	Calculate Smooth L1 Loss(lhs, scalar)
IdentityAttachKLSparseReg	Apply a sparse regularization to the output a sigmoid activation function.
MakeLoss	Get output from a symbol and pass 1 gradient back.
BlockGrad	Get output from a symbol and pass 0 gradient back
Custom	Custom operator implemented in frontend.

Module API

Overview

The module API, defined in the module (or simply mod) package, provides an intermediate and high-level interface for performing computation with a Symbol. One can roughly think a module is a machine which can execute a program defined by a Symbol.

The class module.Module is a commonly used module, which accepts a Symbol as the input:

data = mx.symbol.Variable('data')
fc1  = mx.symbol.FullyConnected(data, name='fc1', num_hidden=128)
act1 = mx.symbol.Activation(fc1, name='relu1', act_type="relu")
fc2  = mx.symbol.FullyConnected(act1, name='fc2', num_hidden=10)
out  = mx.symbol.SoftmaxOutput(fc2, name = 'softmax')
mod = mx.mod.Module(out)  # create a module by given a Symbol

Assume there is a valid MXNet data iterator data. We can initialize the module:

mod.bind(data_shapes=data.provide_data,
         label_shapes=data.provide_label)  # create memory by given input shapes
mod.init_params()  # initial parameters with the default random initializer

Now the module is able to compute. We can call high-level API to train and predict:

mod.fit(data, num_epoch=10, ...)  # train
mod.predict(new_data)  # predict on new data

or use intermediate APIs to perform step-by-step computations

mod.forward(data_batch)  # forward on the provided data batch
mod.backward()  # backward to calculate the gradients
mod.update()  # update parameters using the default optimizer

A detailed tutorial is available at http://mxnet.io/tutorials/python/module.html.

Note

module is used to replace model, which has been deprecated.

The module package provides several modules:

概述	说明
BaseModule	The base class of a module.
Module	Module is a basic module that wrap a Symbol.
SequentialModule	A SequentialModule is a container module that can chain multiple modules together.
BucketingModule	This module helps to deal efficiently with varying-length inputs.
PythonModule	A convenient module class that implements many of the module APIs as empty functions.
PythonLossModule	A convenient module class that implements many of the module APIs as empty functions.

We summarize the interface for each class in the following sections.

The BaseModule class

The BaseModule is the base class for all other module classes. It defines the interface each module class should provide.

Initialize memory

初始化内存	说明
BaseModule.bind	Bind the symbols to construct executors.

Get and set parameters

获取设置参数	说明
BaseModule.init_params	Initialize the parameters and auxiliary states.
BaseModule.set_params	Assign parameter and aux state values.
BaseModule.get_params	Get parameters, those are potentially copies of the the actual parameters used to do computation on the device.
BaseModule.save_params	Save model parameters to file.
BaseModule.load_params	Load model parameters from file.

Train and predict

训练和预测	说明
BaseModule.fit	Train the module parameters.
BaseModule.score	Run prediction on eval_data and evaluate the performance according to eval_metric.
BaseModule.iter_predict	Iterate over predictions.
BaseModule.predict	Run prediction and collect the outputs.

Forward and backward

向前向后	说明
BaseModule.forward	Forward computation.
BaseModule.backward	Backward computation.
BaseModule.forward_backward	A convenient function that calls both forward and backward.

Update parameters

更新参数	说明
BaseModule.init_optimizer	Install and initialize optimizers.
BaseModule.update	Update parameters according to the installed optimizer and the gradients computed in the previous forward-backward batch.
BaseModule.update_metric	Evaluate and accumulate evaluation metric on outputs of the last forward computation.

Input and output

输入与输出	说明
BaseModule.data_names	A list of names for data required by this module.
BaseModule.output_names	A list of names for the outputs of this module.
BaseModule.data_shapes	A list of (name, shape) pairs specifying the data inputs to this module.
BaseModule.label_shapes	A list of (name, shape) pairs specifying the label inputs to this module.
BaseModule.output_shapes	A list of (name, shape) pairs specifying the outputs of this module.
BaseModule.get_outputs	Get outputs of the previous forward computation.
BaseModule.get_input_grads	Get the gradients to the inputs, computed in the previous backward computation.

Others

其它	说明
BaseModule.get_states	Get states from all devices
BaseModule.set_states	Set value for states.
BaseModule.install_monitor	Install monitor on all executors.
BaseModule.symbol	Get the symbol associated with this module.

Other build-in modules

Besides the basic interface defined in BaseModule, each module class supports additional functionality. We summarize them in this section.

Class Module

类模块	说明
Module.load	Create a model from previously saved checkpoint.
Module.save_checkpoint	Save current progress to checkpoint.
Module.reshape	Reshape the module for new input shapes.
Module.borrow_optimizer	Borrow optimizer from a shared module.
Module.save_optimizer_states	Save optimizer (updater) state to file
Module.load_optimizer_states	Load optimizer (updater) state from file

Class BucketModule

BucketModule	说明
BucketModule.switch_bucket

Class SequentialModule

SequentialModule	说明
SequentialModule.add	Add a module to the chain.

KVStore API

Basic Push and Pull

Provides basic operation over multiple devices (GPUs) on a single device.

Initialization

Let’s consider a simple example. It initializes a (int, NDArray) pair into the store, and then pulls the value out.

>>> kv = mx.kv.create('local') # create a local kv store.
>>> shape = (2,3)
>>> kv.init(3, mx.nd.ones(shape)*2)
>>> a = mx.nd.zeros(shape)
>>> kv.pull(3, out = a)
>>> print a.asnumpy()
[[ 2.  2.  2.]
 [ 2.  2.  2.]]

Push, Aggregation, and Updater

For any key that’s been initialized, you can push a new value with the same shape to the key, as follows:

>>> kv.push(3, mx.nd.ones(shape)*8)
>>> kv.pull(3, out = a) # pull out the value
>>> print a.asnumpy()
[[ 8.  8.  8.]
 [ 8.  8.  8.]]

The data that you want to push can be stored on any device. Furthermore, you can push multiple values into the same key, where KVStore first sums all of these values, and then pushes the aggregated value, as follows:

>>> gpus = [mx.gpu(i) for i in range(4)]
>>> b = [mx.nd.ones(shape, gpu) for gpu in gpus]
>>> kv.push(3, b)
>>> kv.pull(3, out = a)
>>> print a.asnumpy()
[[ 4.  4.  4.]
 [ 4.  4.  4.]]

For each push command, KVStore applies the pushed value to the value stored by an updater. The default updater is ASSIGN. You can replace the default to control how data is merged.

>>> def update(key, input, stored):
>>>     print "update on key: %d" % key
>>>     stored += input * 2
>>> kv._set_updater(update)
>>> kv.pull(3, out=a)
>>> print a.asnumpy()
[[ 4.  4.  4.]
 [ 4.  4.  4.]]
>>> kv.push(3, mx.nd.ones(shape))
update on key: 3
>>> kv.pull(3, out=a)
>>> print a.asnumpy()
[[ 6.  6.  6.]
 [ 6.  6.  6.]]

Pull

You’ve already seen how to pull a single key-value pair. Similar to the way that you use the push command, you can pull the value into several devices with a single call.

>>> b = [mx.nd.ones(shape, gpu) for gpu in gpus]
>>> kv.pull(3, out = b)
>>> print b[1].asnumpy()
[[ 6.  6.  6.]
 [ 6.  6.  6.]]

List Key-Value Pairs

All of the operations that we’ve discussed so far are performed on a single key. KVStore also provides the interface for generating a list of key-value pairs. For a single device, use the following:

>>> keys = [5, 7, 9]
>>> kv.init(keys, [mx.nd.ones(shape)]*len(keys))
>>> kv.push(keys, [mx.nd.ones(shape)]*len(keys))
update on key: 5
update on key: 7
update on key: 9
>>> b = [mx.nd.zeros(shape)]*len(keys)
>>> kv.pull(keys, out = b)
>>> print b[1].asnumpy()
[[ 3.  3.  3.]
 [ 3.  3.  3.]]

For multiple devices:

Data Loading API¶

Overview

This document summeries supported data formats and iterator APIs to read the data including

概述	说明
mxnet.io	Data iterators for common data formats.
mxnet.recordio	Read and write for the RecordIO data format.
mxnet.image	Read invidual image files and perform augmentations.

It will also show how to write an iterator for a new data format.

A data iterator reads data batch by batch.

>>> data = mx.nd.ones((100,10))
>>> nd_iter = mx.io.NDArrayIter(data, batch_size=25)
>>> for batch in nd_iter:
...     print(batch.data)
[0)>]
[0)>]
[0)>]
[0)>]

If nd_iter.reset() is called, then reads the data again from beginning.

In addition, an iterator provides information about the batch, including the shapes and name.

>>> nd_iter = mx.io.NDArrayIter(data={'data':mx.nd.ones((100,10))},
...                             label={'softmax_label':mx.nd.ones((100,))},
...                             batch_size=25)
>>> print(nd_iter.provide_data)
[DataDesc[data,(25, 10L),'numpy.float32'>,NCHW]]
>>> print(nd_iter.provide_label)
[DataDesc[softmax_label,(25,),'numpy.float32'>,NCHW]]

So this iterator can be used to train a symbol whose input data variable has name data and input label variable has name softmax_label.

>>> data = mx.sym.Variable('data')
>>> label = mx.sym.Variable('softmax_label')
>>> fullc = mx.sym.FullyConnected(data=data, num_hidden=1)
>>> loss = mx.sym.SoftmaxOutput(data=data, label=label)
>>> mod = mx.mod.Module(loss)
>>> print(mod.data_names)
['data']
>>> print(mod.label_names)
['softmax_label']
>>> mod.bind(data_shapes=nd_iter.provide_data, label_shapes=nd_iter.provide_label)

Then we can call mod.fit(nd_iter, num_epoch=2) to train loss by 2 epochs.

Data iterators

数据迭代器	说明
io.NDArrayIter	Iterating on either mx.nd.NDArray or numpy.ndarray.
io.CSVIter	Iterating on CSV files
io.ImageRecordIter	Iterating on image RecordIO files
io.ImageRecordUInt8Iter	Iterating on image RecordIO files
io.MNISTIter	Iterating on the MNIST dataset.
recordio.MXRecordIO	Read/write RecordIO format data.
recordio.MXIndexedRecordIO	Read/write RecordIO format data supporting random access.
image.ImageIter	Image data iterator with a large number of augmentation choices.

Helper classes and functions

Data structures and other iterators provided in the mxnet.io packages.

帮助类与函数	说明
io.DataDesc	Data description
io.DataBatch	A data batch.
io.DataIter	The base class of a data iterator.
io.ResizeIter	Resize a data iterator to a given number of batches.
io.PrefetchingIter	Performs pre-fetch for other data iterators.
io.MXDataIter	A python wrapper a C++ data iterator.

A list of image modification functions provided by mxnet.image.

mxnet.image	说明
image.imdecode	Decode an image to an NDArray.
image.scale_down	Scale down crop size if it’s bigger than image size.
image.resize_short	Resize shorter edge to size.
image.fixed_crop	Crop src at fixed location, and (optionally) resize it to size.
image.random_crop	Randomly crop src with size.
image.center_crop	Centrally crop src with size.
image.color_normalize	Normalize src with mean and std.
image.random_size_crop	Randomly crop src with size.
image.ResizeAug	Make resize shorter edge to size augmenter.
image.RandomCropAug	Make random crop augmenter
image.RandomSizedCropAug	Make random crop with random resizing and random aspect ratio jitter augmenter.
image.CenterCropAug	Make center crop augmenter.
image.RandomOrderAug	Apply list of augmenters in random order
image.ColorJitterAug	Apply random brightness, contrast and saturation jitter in random order.
image.LightingAug	Add PCA based noise.
image.ColorNormalizeAug	Mean and std normalization.
image.HorizontalFlipAug	Random horizontal flipping.
image.CastAug	Cast to float32
image.CreateAugmenter	Creates an augmenter list.

Functions to read and write RecordIO files.

RecordIO	说明
recordio.pack	Pack a string into MXImageRecord.
recordio.unpack	Unpack a MXImageRecord to string.
recordio.unpack_img	Unpack a MXImageRecord to image.
recordio.pack_img	Pack an image into MXImageRecord.

Develop a new iterator

Writing a new data iterator in Python is straightforward. Most MXNet training/inference program accepts an iteratable object with provide_data and provide_label properties. This tutorial how to write an iterator from scratch.

The following example demonstrates how to combine multiple data iterators into a single one. It can be used for multiple modality training such as image captioning, in which images can be read byImageRecordIter while documents by CSVIter

class MultiIter:
    def __init__(self, iter_list):
        self.iters = iter_list
    def next(self):
        batches = [i.next() for i in self.iters]
        return DataBatch(data=[*b.data for b in batches],
                         label=[*b.label for b in batches])
    def reset(self):
        for i in self.iters:
            i.reset()
    @property
    def provide_data(self):
        return [*i.provide_data for i in self.iters]
    @property
    def provide_label(self):
        return [*i.provide_label for i in self.iters]

iter = MultiIter([mx.io.ImageRecordIter('image.rec'), mx.io.CSVIter('txt.csv')])

Parsing and another pre-processing such as augmentation may be expensive. If performance is critical, we can implement a data iterator in C++. Refer to src/io for examples.

Optimization: initialize and update weights

Overview

This document summaries the APIs used to initialize and update the model weights during training

概述	说明
mxnet.initializer	Weight initializer.
mxnet.optimizer	Weight updating functions.
mxnet.lr_scheduler	Scheduling learning rate.

and how to develop a new optimization algorithm in MXNet.

Assume there there is a pre-defined Symbol and a Module is created for it

>>> data = mx.symbol.Variable('data')
>>> label = mx.symbol.Variable('softmax_label')
>>> fc = mx.symbol.FullyConnected(data, name='fc', num_hidden=10)
>>> loss = mx.symbol.SoftmaxOutput(fc, label, name='softmax')
>>> mod = mx.mod.Module(loss)
>>> mod.bind(data_shapes=[('data', (128,20))], label_shapes=[('softmax_label', (128,))])

Next we can initialize the weights with values sampled uniformly from [-1,1]:

>>> mod.init_params(mx.initializer.Uniform(scale=1.0))

Then we will train a model with standard SGD which decreases the learning rate by multiplying 0.9 for each 100 batches.

>>> lr_sch = mx.lr_scheduler.FactorScheduler(step=100, factor=0.9)
>>> mod.init_optimizer(
...     optimizer='sgd', optimizer_params=(('learning_rate', 0.1), ('lr_scheduler', lr_sch)))

Finally run mod.fit(...) to start training.

The mxnet.initializer package

The base class Initializer defines the default behaviors to initialize various parameters, such as set bias to 1, except for the weight. Other classes then defines how to initialize the weight.

mxnet.initializer	说明
Initializer	The base class of an initializer.
Uniform	Initialize the weight with value uniformly sampled from [-scale, scale].
Normal	Initialize the weight with value sampled according to normal(0, sigma).
Load	Initialize by loading data from file or dict.
Mixed	Initialize parameters using multiple initializers.
Zero	Initialize the weight to 0.
One	Initialize the weight to 1.
Constant	Initialize the weight to a scalar value.
Orthogonal	Initialize weight as orthogonal matrix.
Xavier	Initialize the weight with Xavier or other similar schemes.
MSRAPrelu	Initialize the weight according to a MSRA paper.
Bilinear	Initialize weight for upsampling layers.
FusedRNN	Initialize parameters for fused rnn layers.

The mxnet.optimizer package

The base class Optimizer accepts commonly shared arguments such as learning_rate and defines the interface. Each other class in this package implements one weight updating function.

mxnet.optimizer	说明
Optimizer	The base class inherited by all optimizers.
SGD	The SGD optimizer with momentum and weight decay.
NAG	Nesterov accelerated SGD.
RMSProp	The RMSProp optimizer.
Adam	The Adam optimizer.
AdaGrad	AdaGrad optimizer
AdaDelta	The AdaDelta optimizer.
DCASGD	The DCASGD optimizer
SGLD	Stochastic Gradient Riemannian Langevin Dynamics.

The mxnet.lr_scheduler package

The base class LRScheduler defines the interface, while other classes implement various schemes to change the learning rate during training.

mxnet.lr_scheduler	说明
LRScheduler	Base class of a learning rate scheduler.
FactorScheduler	Reduce the learning rate by a factor for every n steps.
MultiFactorScheduler	Reduce the learning rate by given a list of steps.

Implement a new algorithm

Most classes listed in this document are implemented in Python by using NDArray. So implementing new weight updating or initialization functions is straightforward.

For initializer, create a subclass of Initializer and define the _init_weight method. We can also change the default behaviors to initialize other parameters such as _init_bias. Seeinitializer.py for examples.

For optimizer, create a subclass of Optimizer and implement two methods create_state and update. Also add @mx.optimizer.Optimizer.register before this class. See optimizer.py for examples.

For lr_scheduler, create a subclass of LRScheduler and then implement the __call__ method. See lr_scheduler.py for examples.

附录

官方链接